大数据：spark集群搭建

创建spark用户组，组ID1000

groupadd -g 1000 spark

在spark用户组下创建用户ID 2000的spark用户 获取视频中文档资料及完整视频的伙伴请加QQ群：947967114
useradd -u 2000 -g spark spark

设置密码

passwd spark

修改sudo权限

chmod u+w /etc/sudoers

vi /etc/sudoers

找到

root ALL=(ALL) ALL

添加

spark ALL=(ALL) ALL

创建一个app目录用来存放spark的软件环境（jdk、scala、spark）

mkdir /app

修改这个文件的属组和属主

chown -R spark:spark /app

创建soft

mkdir /app/soft

创建spark

mkdir /app/spark

创建/spark/work

mkdir -p /home/spark/work

改变/spark/work属组和属主

chown -R spark:spark /home/spark/work

切换用户

su root

解压JDK

cd /tmp/

tar zxvf jdk-8u192-linux-x64.tar.gz -C /app/soft/

如果没有权限首先使用chmod 777 -R /tmp修改权限

cd /app/soft/

ll -a

配置/etc/profile

sudo vi /etc/profile，所有需要的配置都添加了

JAVA_HOME=/app/soft/jdk1.8.0_192

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin

export PATH

让配置生效 获取视频中文档资料及完整视频的伙伴请加QQ群：947967114
source /etc/profile

安装scala：

tar zxvf /tmp/scala-2.11.12.tgz -C /app/soft/

配置环境变量

sudo vi /etc/profile

JAVA_HOME=/app/soft/jdk1.8.0_192

SCALA_HOME=/app/soft/scala-2.11.12/

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin

export PATH

配置ssh无秘登录

ssh-keygen -t rsa

cd ~/

cd .ssh/

修改公钥的名字

master节点：mv id_rsa.pub authorized_keys_master.pub

slave1节点：mv id_rsa.pub authorized_keys_slave1.pub

slave2节点：mv id_rsa.pub authorized_keys_slave2.pub

把slave1和slave2的公钥给master

slave1节点：scp authorized_keys_slave1.pub [email protected]:/home/spark/.ssh/

slave2节点：scp authorized_keys_slave2.pub [email protected]:/home/spark/.ssh/

把三个节点的公钥都写在一个文件中

cat authorized_keys_master.pub >> authorized_keys

cat authorized_keys_slave1.pub >> authorized_keys

cat authorized_keys_slave2.pub >> authorized_keys

查看一下总的公钥文件

vi authorized_keys

把总的公钥文件authorized_keys给到slave1和slave2节点

scp authorized_keys [email protected]:/home/spark/.ssh

scp authorized_keys [email protected]:/home/spark/.ssh

修改authorized_keys的操作权限，三个节点都需要修改

chmod 400 authorized_keys

验证免密登录是否成功

ssh master

ssh slave1

ssh slave2

ssh master

安装spark：

tar -zxf /tmp/spark-2.1.0-bin-hadoop2.6.gz -C /app/spark/

cd /app/spark/

ls

cd spark-2.1.0-bin-hadoop2.6/

配置环境变量：

vi /etc/profile

JAVA_HOME=/app/soft/jdk1.8.0_192

SCALA_HOME=/app/soft/scala-2.11.12/

SPARK_HOME=/app/spark/spark-2.1.0-bin-hadoop2.6

PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin

export PATH

配置spark的核心文件：

cd spark-2.1.0-bin-hadoop2.6/

cd conf/

配置slaves

mv slaves.template slaves

vi slaves 添加三个节点

master

slave1

slave2

配置spark-env.sh

cp spark-env.sh.template spark-env.sh

vi spark-env.sh

export JAVA_HOME=/app/soft/jdk1.8.0_192

export SCALA_HOME=/app/soft/scala-2.11.12

export SPARK_MASTER_IP=master

export SPARK_MASTER_PORT=7077

export SPARK_EXECUTOR_INSTANCES=1

export SPARK_WORKER_INSTANCES=1

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=1024M

export SPARK_MASTER_WEBUI=8080

export SPARK_CONF_DIR=/app/spark/spark-2.1.0-bin-hadoop2.6/conf/

把所有的节点的app的work和soft权限都改成777：在所有的节点上执行 chmod 777 -R /app/soft 和chmod 777 -R /app/spark

scp -r /app/spark/ [email protected]:/app/

scp -r /app/soft/ [email protected]:/app/

到此spark集群已经搭建完成：

开启：start-all.sh获取视频中文档资料及完整视频的伙伴请加QQ群：947967114
jps可以看到如下进程：

master节点：

3617 Worker

3507 Master

4156 Jps

slave1节点：

3361 Worker

3702 Jps

slave2节点：

3319 Worker

3647 Jps

开启spark-shell验证：

spark-shell --master spark://master:7077 --executor-memory 1024m --driver-memory 1024m

启动之后会显示如下内容：

18/11/29 16:13:46 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

18/11/29 16:13:47 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

Spark context Web UI available at http://192.168.0.10:4040

Spark context available as ‘sc‘ (master = spark://master:7077, app id = app-20181129161336-0000).

Spark session available as ‘spark‘.

Welcome to

  ____              __

 / __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  ‘_/

// ./_,// //_\ version 2.1.0

  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_192)

Type in expressions to have them evaluated.

Type :help for more information.

scala>

就可以在>后面书写spark代码了：

g NoSuchObjectException

Spark context Web UI available at http://192.168.0.10:4040

Spark context available as ‘sc‘ (master = spark://master:7077, app id = app-20181129161336-0000).

Spark session available as ‘spark‘.

Welcome to

  ____              __

 / __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  ‘_/

// ./_,// //_\ version 2.1.0

  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_192)

Type in expressions to have them evaluated.

Type :help for more information.

scala> sc.textFile("/app/spark/spark-2.1.0-bin-hadoop2.6/README.md").flatMap(.split(" ")).map(x=>(x,1)).reduceByKey(+_).map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1)).take(10)

res0: Array[(String, Int)] = Array(("",71), (the,24), (to,17), (Spark,16), (for,12), (and,9), (##,8), (a,8), (can,7), (run,7))

scala>获取视频中文档资料及完整视频的伙伴请加QQ群：947967114

原文地址：http://blog.51cto.com/14043271/2324690