spark过程

scala配置

1、下载解压包

tar -xvf scala-2.10.4.tgz -C /usr/local/

2、包重命名为scala

3、配置环境变量

export SCALA_HOME=/usr/local/scala

export PATH=$PATH:/usr/local/scala/bin

4、执行生效source /etc/profile

##验证配置

scala -version 得到

Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL

如果得到以上这句话，恭喜你，scala配置成功！

maven配置

1、下载解压包

tar -xvf apache-maven-3.3.9-bin.tar.gz -C /usr/local/

2、包重命名为maven

3、配置环境变量/etc/profile

export MAVEN_HOME=/usr/local/maven

export PATH=$PATH:/usr/local/maven/bin

export MAVEN_OPTS="-Xms256m -Xmx512m"

##验证配置

mvn -v 得到

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: /usr/local/maven

Java version: 1.7.0_55, vendor: Oracle Corporation

Java home: /usr/local/jdk/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "i386", family: "unix"

如果得到以上这句话，恭喜你，scala配置成功！

安装编译spark

1、解压源码包：tar -zxvf spark-2.0.2-bin-hadoop2.7.tgz -C /usr/local/

cd /usr/local/

mv spark-2.0.2-bin-hadoop2.7 spark-2.0.2

source /etc/profile

2、复制配置模板文件

cd /usr/local/spark-2.0.2/conf

cp spark-env.sh.template spark-env.sh

cp slaves.template slaves

cp spark-defaults.conf.template spark-defaults.conf

主要配置JAVA_HOME、SCALA_HOME、HADOOP_HOME、HADOOP_CONF_DIR、SPARK_MASTER_IP等

vim spark-env.sh

export JAVA_HOME=/usr/local/jdk

export SCALA_HOME=/usr/local/scala

export HADOOP_HOME=/usr/local/hadoop

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

export SPARK_LAUNCH_WITH_SCALA=0

export SPARK_WORKER_MEMORY=1g

export SPARK_DRIVER_MEMORY=1g

export SPARK_MASTER_IP=192.168.1.114

export SPARK_LIBRARY_PATH=/usr/local/spark-2.0.2/lib

export SPARK_MASTER_WEBUI_PORT=18080

export SPARK_WORKER_DIR=/home/spark

export SPARK_MASTER_PORT=7077

export SPARK_WORKER_PORT=7078

export SPARK_LOG_DIR=/home/spark_log

export SPARK_PID_DIR=‘/home/spark/run‘

slaves（将所有节点都加入，master节点同时也是worker节点）

spark-defaults.conf

spark.master                     yarn-client

spark.home                       /root/spark-without-hive

spark.eventLog.enabled           true

spark.eventLog.dir               hdfs://Goblin01:8020/spark-log

spark.serializer                 org.apache.spark.serializer.KryoSerializer

spark.executor.memory            1g

spark.driver.memory              1g

spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.master指定Spark运行模式，可以是yarn-client、yarn-cluster...

spark.home指定SPARK_HOME路径

spark.eventLog.enabled需要设为true

spark.eventLog.dir指定路径，放在master节点的hdfs中，端口要跟hdfs设置的端口一致（默认为8020），否则会报错

spark.executor.memory和spark.driver.memory指定executor和dirver的内存，512m或1g，既不能太大也不能太小，因为太小运行不了，太大又会影响其他服务

配置yar-site.xml,跟hdfs-site.xml在同一个路径下（$HADOOP_HOME/etc/hadoop)

ll /usr/local/hadoop/etc/hadoop/yarn-site.xml

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>haproxy:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>haproxy:8035</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>mycat:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>mycat:8088</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>

</property>

</configuration>

把spark-2.0.2复制到其他节点

启动start-all.sh

7. 运行

1) 准备一个文本文件放在/logs/wordcount.log内容为:

2) 运行spark-shell