大数据学习环境搭建(CentOS6.9+Hadoop2.7.3+Hive1.2.1+Hbase1.3.1+Spark2.1.1)


node1

192.168.1.11
node2

192.168.1.12
node3

192.168.1.13

 备注

NameNode
Hadoop
Y
Y

 高可用

DateNode
Y
Y
Y

ResourceManager
Y
Y

 高可用
NodeManager
Y
Y
Y

JournalNodes
Y
Y
Y

 奇数个,至少3个节点
ZKFC（DFSZKFailoverController）
Y
Y

 有namenode的地方就有ZKFC

QuorumPeerMain
Zookeeper
Y
Y
Y

MySQL
HIVE
Y

 Hive元数据库

Metastore（RunJar）
Y

HIVE（RunJar）
Y
HMaster HBase Y Y 高可用
HRegionServer Y Y Y

Spark（Master）
Spark
Y
Y

 高可用

Spark（Worker）
Y
Y
Y

以前搭建过一套，带Federation，至少需4台机器，过于复杂，笔记本也吃不消。现为了学习Spark2.0版本，决定去掉Federation，简化学习环境，不过还是完全分布式

所有软件包：

apache-ant-1.9.9-bin.tar.gz

apache-hive-1.2.1-bin.tar.gz

apache-maven-3.3.9-bin.tar.gz

apache-tomcat-6.0.44.tar.gz

CentOS-6.9-x86_64-minimal.iso

findbugs-3.0.1.tar.gz

hadoop-2.7.3-src.tar.gz

hadoop-2.7.3.tar.gz

hadoop-2.7.3（自已编译的centOS6.9版本）.tar.gz

hbase-1.3.1-bin(自己编译).tar.gz

hbase-1.3.1-src.tar.gz

jdk-8u121-linux-x64.tar.gz

mysql-connector-java-5.6-bin.jar

protobuf-2.5.0.tar.gz

scala-2.11.11.tgz

snappy-1.1.3.tar.gz

spark-2.1.1-bin-hadoop2.7.tgz

关闭防火墙

[[email protected] ~]# service iptables stop

[[email protected] ~]# chkconfig iptables off

zookeeper

[[email protected] ~]# wget -O /root/zookeeper-3.4.9.tar.gz https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

[[email protected] ~]# tar -zxvf /root/zookeeper-3.4.9.tar.gz -C /root

[[email protected] ~]# cp /root/zookeeper-3.4.9/conf/zoo_sample.cfg /root/zookeeper-3.4.9/conf/zoo.cfg

[[email protected] ~]# vi /root/zookeeper-3.4.9/conf/zoo.cfg

[[email protected] ~]# vi /root/zookeeper-3.4.9/bin/zkEnv.sh

[[email protected] ~]# mkdir /root/zookeeper-3.4.9/logs

[[email protected] ~]# vi /root/zookeeper-3.4.9/conf/log4j.properties

[[email protected] ~]# mkdir /root/zookeeper-3.4.9/zkData

[[email protected] ~]# scp -r /root/zookeeper-3.4.9 node2:/root

[[email protected] ~]# scp -r /root/zookeeper-3.4.9 node3:/root

[[email protected] ~]# touch /root/zookeeper-3.4.9/zkData/myid

[[email protected] ~]# echo 1 > /root/zookeeper-3.4.9/zkData/myid

[[email protected] ~]# touch /root/zookeeper-3.4.9/zkData/myid

[[email protected] ~]# echo 2 > /root/zookeeper-3.4.9/zkData/myid

[[email protected] ~]# touch /root/zookeeper-3.4.9/zkData/myid

[[email protected] ~]# echo 3 > /root/zookeeper-3.4.9/zkData/myid

环境变量

[[email protected] ~]# vi /etc/profile

export JAVA_HOME=/root/jdk1.8.0_121

export SCALA_HOME=/root/scala-2.11.11

export HADOOP_HOME=/root/hadoop-2.7.3

export HIVE_HOME=/root/apache-hive-1.2.1-bin

export HBASE_HOME=/root/hbase-1.3.1

export SPARK_HOME=/root/spark-2.1.1-bin-hadoop2.7

export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:/root:$HIVE_HOME/bin:$HBASE_HOME/bin:$SPARK_HOME

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

[[email protected] ~]# source /etc/profile

[[email protected] ~]# scp /etc/profile node2:/etc

[[email protected] ~]# source /etc/profile

[[email protected]~]# scp /etc/profile node3:/etc

[[email protected] ~]# source /etc/profile

Hadoop

[[email protected] ~]# wget -O /root/hadoop-2.7.3.tar.gz http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

[[email protected] ~]# tar -zxvf /root/hadoop-2.7.3.tar.gz -C /root

[[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/hadoop-env.sh



 

[[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/hdfs-site.xml



<property>



   <name>dfs.replication</name>



   <value>2</value>



</property>



<property>



   <name>dfs.blocksize</name>



   <value>64m</value>



</property>



<property>



   <name>dfs.permissions.enabled</name>



<value>false</value>



</property>

<property>

  <name>dfs.nameservices</name>

  <value>mycluster</value>

</property>

<property>

  <name>dfs.ha.namenodes.mycluster</name>

  <value>nn1,nn2</value>

</property>

<property>

  <name>dfs.namenode.rpc-address.mycluster.nn1</name>

  <value>node1:8020</value>

</property>

<property>

  <name>dfs.namenode.rpc-address.mycluster.nn2</name>

  <value>node2:8020</value>

</property>

<property>

  <name>dfs.namenode.http-address.mycluster.nn1</name>

  <value>node1:50070</value>

</property>

<property>

  <name>dfs.namenode.http-address.mycluster.nn2</name>

  <value>node2:50070</value>

</property>

<property>

  <name>dfs.namenode.shared.edits.dir</name>

  <value>qjournal://node1:8485;node2:8485;node3:8485/mycluster</value>

</property>

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/root/hadoop-2.7.3/tmp/journal</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled.mycluster</name>

<value>true</value>

</property>

<property>

  <name>dfs.client.failover.proxy.provider.mycluster</name>

  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

  <name>dfs.ha.fencing.methods</name>

  <value>sshfence</value>

</property>

<property>

  <name>dfs.ha.fencing.ssh.private-key-files</name>

  <value>/root/.ssh/id_rsa</value>

</property>

[[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/core-site.xml

<property>

<name>fs.defaultFS</name>

<value>hdfs://mycluster</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/root/hadoop-2.7.3/tmp</value>

</property>

<property>

<name>ha.zookeeper.quorum</name>

<value>node1:2181,node2:2181,node3:2181</value>

</property>

[[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/slaves

node1

node2

node3

[[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/yarn-env.sh



 [[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/mapred-site.xml

<configuration>

<property> 

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>node1:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>node1:19888</value>

</property>

<property>

<name>mapreduce.jobhistory.max-age-ms</name>

<value>6048000000</value>

</property>

</configuration>

[[email protected] ~]# vi /root/hadoop-2.7.3/etc/hadoop/yarn-site.xml

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>                                                                

<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.ha.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.cluster-id</name>

<value>yarn-cluster</value>

</property>

<property>

<name>yarn.resourcemanager.ha.rm-ids</name>

<value>rm1,rm2</value>

</property>

<property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>node1</value>

</property>

<property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>node2</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address.rm1</name>

<value>node1:8088</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address.rm2</name>

<value>node2:8088</value>

</property>

<property>

<name>yarn.resourcemanager.zk-address</name>

<value>node1:2181,node2:2181,node3:2181</value>

</property>

<property>

<name>yarn.resourcemanager.recovery.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.store.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

<property>

<name>yarn.log-aggregation-enable</name>   

<value>true</value>

</property>

<property>

<name>yarn.log.server.url</name>

<value>http://node1:19888/jobhistory/logs</value>

</property>

[[email protected] ~]# mkdir -p /root/hadoop-2.7.3/tmp/journal

[[email protected] ~]# mkdir -p /root/hadoop-2.7.3/tmp/journal

[[email protected] ~]# mkdir -p /root/hadoop-2.7.3/tmp/journal

将编译的本地包中的native库替换/root/hadoop-2.7.3/lib/native

[[email protected] ~]# scp -r /root/hadoop-2.7.3/ node2:/root

[[email protected] ~]# scp -r /root/hadoop-2.7.3/ node3:/root

查看自己的Hadoop是32位还是64位

[[email protected] native]# file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

[[email protected] native]# pwd

/root/hadoop-2.7.3/lib/native

启动ZK

[[email protected] ~]#/root/zookeeper-3.4.9/bin/zkServer.sh start

[[email protected] ~]#/root/zookeeper-3.4.9/bin/zkServer.sh start

[[email protected] ~]#/root/zookeeper-3.4.9/bin/zkServer.sh start

格式化zkfc

[[email protected] ~]# /root/hadoop-2.7.3/bin/hdfs zkfc -formatZK

[[email protected] ~]# /root/zookeeper-3.4.9/bin/zkCli.sh

启动journalnode

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start journalnode

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start journalnode

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start journalnode

Namenode格式化和启动

[[email protected] ~]# /root/hadoop-2.7.3/bin/hdfs namenode -format

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start namenode

[[email protected] ~]# /root/hadoop-2.7.3/bin/hdfs namenode -bootstrapStandby

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start namenode

启动zkfc

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start zkfc

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start zkfc

启动datanode

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start datanode

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start datanode

[[email protected] ~]# /root/hadoop-2.7.3/sbin/hadoop-daemon.sh start datanode

启动yarn

[[email protected] ~]# /root/hadoop-2.7.3/sbin/yarn-daemon.sh start resourcemanager

[[email protected] ~]# /root/hadoop-2.7.3/sbin/yarn-daemon.sh start resourcemanager

[[email protected] ~]# /root/hadoop-2.7.3/sbin/yarn-daemon.sh start nodemanager

[[email protected] ~]# /root/hadoop-2.7.3/sbin/yarn-daemon.sh start nodemanager

[[email protected] ~]# /root/hadoop-2.7.3/sbin/yarn-daemon.sh start nodemanager

[[email protected] ~]# hdfs dfs -chmod -R 777 /

安装MySQL

[[email protected] ~]# yum remove -y mysql-libs

[[email protected] ~]# yum install mysql-server

[[email protected] ~]# service mysqld start

[[email protected] ~]# chkconfig mysqld on

[[email protected] ~]# mysqladmin -u root password ‘AAAaaa111‘

[[email protected] ~]# mysqladmin -u root -h node1 password ‘AAAaaa111‘

[[email protected] ~]# mysql -h localhost -u root -p

Enter password: AAAaaa111

mysql> GRANT ALL PRIVILEGES ON *.* TO ‘root‘@‘%‘ IDENTIFIED BY ‘AAAaaa111‘ WITH GRANT OPTION;

mysql> flush privileges;

[[email protected] ~]# vi /etc/my.cnf

[client]

default-character-set=utf8

[mysql]

default-character-set=utf8

[mysqld]

character-set-server=utf8

lower_case_table_names = 1

[[email protected] ~]# service mysqld restart

HIVE安装

由于官方提供的spark-2.1.1-bin-hadoop2.7.tgz包中集成的Hive是1.2.1，所以Hive版本选择1.2.1


[[email protected] ~]# wget http://archive.apache.org/dist/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz

[[email protected] ~]# tar -xvf apache-hive-1.2.1-bin.tar.gz

将mysql-connector-java-5.6-bin.jar 驱动放在 /root/hive-1.2.1/lib/ 目录下面

[[email protected] ~]# cp /root/apache-hive-1.2.1-bin/conf/hive-env.sh.template /root/apache-hive-1.2.1-bin/conf/hive-env.sh

[[email protected] ~]# vi /root/apache-hive-1.2.1-bin/conf/hive-env.sh

    export HADOOP_HOME=/root/hadoop-2.7.3

[[email protected] ~]# cp /root/apache-hive-1.2.1-bin/conf/hive-log4j.properties.template /root/apache-hive-1.2.1-bin/conf/hive-log4j.properties

[[email protected] ~]# vi /root/apache-hive-1.2.1-bin/conf/hive-site.xml

<configuration>

<property>

<name>hive.metastore.uris</name>

<value>thrift://node1:9083</value>

</property> 


<property>

<name>hive.metastore.warehouse.dir</name>

<value>/user/hive/warehouse</value>

</property>


<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://node1:3306/hive?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8</value>

</property>


<property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>


<property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>root</value>

</property>


<property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>AAAaaa111</value>

</property>

</configuration>

[[email protected] ~]# vi /etc/init.d/hive-metastore

 /root/apache-hive-1.2.1-bin/bin/hive --service metastore >/dev/null 2>&1 &

[[email protected] ~]# chmod 777 /etc/init.d/hive-metastore

[[email protected] ~]# ln -s /etc/init.d/hive-metastore /etc/rc.d/rc3.d/S65hive-metastore

[[email protected] ~]# hive

[[email protected] ~]# mysql -h localhost -u root -p

mysql> alter database hive character set latin1;

Hbase编译安装

http://archive.apache.org/dist/hbase/1.3.1/hbase-1.3.1-src.tar.gz

官方提供的是基础Hadoop2.5.1编译的，所以要进行编译：

将pom.xml文件中依赖的hadoop版本修改：

<hadoop-two.version>2.5.1</hadoop-two.version>

修改为

<hadoop-two.version>2.7.3</hadoop-two.version>

<compileSource>1.7</compileSource>

修改为

<compileSource>1.8</compileSource>

例如如下命令打包：

mvn clean package -DskipTests -Prelease assembly:single

/root/hbase-1.3.1/hbase-assembly/target/hbase-1.3.1-bin.tar.gz

下面基于此安装包进行Hbase的安装：

[[email protected] ~]# cp /root/hadoop-2.7.3/etc/hadoop/hdfs-site.xml /root/hadoop-2.7.3/etc/hadoop/core-site.xml /root/hbase-1.3.1/conf/

[[email protected] ~]# vi /root/hbase-1.3.1/conf/hbase-env.sh

export JAVA_HOME=/root/jdk1.8.0_121

export HBASE_MANAGES_ZK=false

[[email protected] ~]# vi /root/hbase-1.3.1/conf/hbase-site.xml

<property>

        <name>hbase.rootdir</name>

        <value>hdfs://mycluster:8020/hbase</value>

</property>

<property>

        <name>hbase.cluster.distributed</name>

        <value>true</value>

</property>

<property>

        <name>hbase.zookeeper.quorum</name>

        <value>node1:2181,node2:2181,node3:2181</value>

</property>

<property>

    <name>hbase.master.port</name>

    <value>60000</value>

</property>

<property>

    <name>hbase.master.info.port</name>

    <value>60010</value>

</property>

<property>

    <name>hbase.tmp.dir</name>

    <value>/root/hbase-1.3.1/tmp</value>

</property>

<property>

    <name>hbase.regionserver.port</name>

    <value>60020</value>

</property>

<property>

    <name>hbase.regionserver.info.port</name>

    <value>60030</value>

</property>

[[email protected] ~]# vi /root/hbase-1.3.1/conf/regionservers

node1

node2

node3

[[email protected] ~]# mkdir -p /root/hbase-1.3.1/tmp

[[email protected] ~]# vi /root/hbase-1.3.1/conf/hbase-env.sh

# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+

#export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

#export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

将etc/profile，及hbase复制到其他两个节点上

[[email protected] ~]# start-hbase.sh

#back-master需要手动起

[[email protected] ~]# hbase-daemon.sh start master

[[email protected] ~]# hbase shell

spark

https://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-hadoop2.7.tgz

[[email protected] ~]# cp /root/spark-2.1.1-bin-hadoop2.7/conf/spark-env.sh.template /root/spark-2.1.1-bin-hadoop2.7/conf/spark-env.sh

[[email protected] ~]# vi /root/spark-2.1.1-bin-hadoop2.7/conf/spark-env.sh

export SCALA_HOME=/root/scala-2.11.11

export JAVA_HOME=/root/jdk1.8.0_121

export HADOOP_HOME=/root/hadoop-2.7.3

export HADOOP_CONF_DIR=/root/hadoop-2.7.3/etc/hadoop

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1:2181,node2:2181,node3:2181 -Dspark.deploy.zookeeper.dir=/spark"

[[email protected] ~]# cp /root/spark-2.1.1-bin-hadoop2.7/conf/slaves.template /root/spark-2.1.1-bin-hadoop2.7/conf/slaves

[[email protected] ~]# vi /root/spark-2.1.1-bin-hadoop2.7/conf/slaves

node1

node2

node3

[[email protected] ~]# scp -r /root/spark-2.1.1-bin-hadoop2.7 node2:/root

[[email protected] ~]# scp -r /root/spark-2.1.1-bin-hadoop2.7 node3:/root

[[email protected] ~]# /root/spark-2.1.1-bin-hadoop2.7/sbin/start-all.sh

./start.sh

/root/zookeeper-3.4.9/bin/zkServer.sh start

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/zookeeper-3.4.9/bin/zkServer.sh start‘

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/zookeeper-3.4.9/bin/zkServer.sh start‘


/root/hadoop-2.7.3/sbin/start-dfs.sh

/root/hadoop-2.7.3/sbin/start-yarn.sh

#如果Yarn做HA，则打开

#ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hadoop-2.7.3/sbin/yarn-daemon.sh start resourcemanager‘


/root/hadoop-2.7.3/sbin/hadoop-daemon.sh start zkfc

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hadoop-2.7.3/sbin/hadoop-daemon.sh start zkfc‘

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hadoop-2.7.3/sbin/hadoop-daemon.sh start zkfc‘


/root/hadoop-2.7.3/bin/hdfs haadmin -ns mycluster -failover nn2 nn1

echo ‘Y‘ | ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hadoop-2.7.3/bin/yarn rmadmin -transitionToActive --forcemanual rm1‘


/root/hbase-1.3.1/bin/start-hbase.sh

#如果HBase做HA，则打开

#ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hbase-1.3.1/bin/hbase-daemon.sh start master‘


/root/spark-2.1.1-bin-hadoop2.7/sbin/start-all.sh

#如果Spark做HA，则打开

#ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/spark-2.1.1-bin-hadoop2.7/sbin/start-master.sh‘


/root/hadoop-2.7.3/sbin/mr-jobhistory-daemon.sh start historyserver


echo ‘--------------node1---------------‘

jps | grep -v Jps | sort  -k 2 -t ‘ ‘

echo ‘--------------node2---------------‘

ssh [email protected] "export PATH=/usr/bin:$PATH;jps | grep -v Jps | sort  -k 2 -t ‘ ‘"

echo ‘--------------node3---------------‘

ssh [email protected] "export PATH=/usr/bin:$PATH;jps | grep -v Jps | sort  -k 2 -t ‘ ‘"

./stop.sh

/root/spark-2.1.1-bin-hadoop2.7/sbin/stop-all.sh


/root/hbase-1.3.1/bin/stop-hbase.sh


#如果Yarn开HA，则去掉注释

#ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hadoop-2.7.3/sbin/yarn-daemon.sh stop resourcemanager‘

/root/hadoop-2.7.3/sbin/stop-yarn.sh

/root/hadoop-2.7.3/sbin/stop-dfs.sh


/root/hadoop-2.7.3/sbin/hadoop-daemon.sh stop zkfc

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/hadoop-2.7.3/sbin/hadoop-daemon.sh stop zkfc‘


/root/zookeeper-3.4.9/bin/zkServer.sh stop

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/zookeeper-3.4.9/bin/zkServer.sh stop‘

ssh [email protected] ‘export BASH_ENV=/etc/profile;/root/zookeeper-3.4.9/bin/zkServer.sh stop‘


/root/hadoop-2.7.3/sbin/mr-jobhistory-daemon.sh stop historyserver

./shutdown.sh

ssh [email protected] "export PATH=/usr/bin:$PATH;shutdown -h now"

ssh [email protected] "export PATH=/usr/bin:$PATH;shutdown -h now"

shutdown -h now

./reboot.sh

ssh [email protected] "export PATH=/usr/bin:$PATH;reboot"

ssh [email protected] "export PATH=/usr/bin:$PATH;reboot"

reboot

时间： 2024-10-01 04:55:50

大数据学习环境搭建(CentOS6.9+Hadoop2.7.3+Hive1.2.1+Hbase1.3.1+Spark2.1.1)的相关文章

虚拟机CentOs的安装及大数据的环境搭建

  大数据问题汇总     1.安装问题        1.安装步骤,详见文档<centos虚拟机安装指南>        2.vi编辑器使用问题,详见文档<linux常用命令.pdf>        3.网络中继更改问题        命令:   vi /etc/sysconfig/network-scripts/ifcfg-eth0·········需要修改的代码·········DEVICE=eth0            HWADDR=00:0C:29:11:02:E8  

cdh大数据平台环境搭建片

环境准备 三台服务器(由于我是学习环境,所以三台服务器不属于同一内网),配置如下: 公网ip 119.29.186.83   内网ip 10.104.157.113 公网ip 119.29.250.47   内网ip 10.104.9.181 公网ip 119.29.251.99   内网ip 10.104.196.48 软件包下载: http://archive.cloudera.com/cdh5/cdh/5/  下载地址 下载的文件名: 1.hadoop-2.6.0-cdh5.8.0.tar

大数据学习系列之四 ----- Hadoop+Hive环境搭建图文详解(单机)

引言 在大数据学习系列之一 ----- Hadoop环境搭建(单机) 成功的搭建了Hadoop的环境,在大数据学习系列之二 ----- HBase环境搭建(单机)成功搭建了HBase的环境以及相关使用介绍.本文主要讲解如何搭建Hadoop+Hive的环境. 一.环境准备 1,服务器选择 本地虚拟机 操作系统:linux CentOS 7 Cpu:2核 内存:2G 硬盘:40G 说明:因为使用阿里云服务器每次都要重新配置,而且还要考虑网络传输问题,于是自己在本地便搭建了一个虚拟机,方便文件的传输以

大数据学习系列之六 ----- Hadoop+Spark环境搭建

引言 在上一篇中 大数据学习系列之五 ----- Hive整合HBase图文详解 : http://www.panchengming.com/2017/12/18/pancm62/ 中使用Hive整合HBase,并且测试成功了.在之前的大数据学习系列之一 ----- Hadoop环境搭建(单机) : http://www.panchengming.com/2017/11/26/pancm55/ 中成功的搭建了Hadoop的环境,本文主要讲的是Hadoop+Spark 的环境.虽然搭建的是单机版,

大数据学习系列之七 ----- Hadoop+Spark+Zookeeper+HBase+Hive集群搭建 图文详解

引言 在之前的大数据学习系列中,搭建了Hadoop+Spark+HBase+Hive 环境以及一些测试.其实要说的话,我开始学习大数据的时候,搭建的就是集群,并不是单机模式和伪分布式.至于为什么先写单机的搭建,是因为作为个人学习的话,单机已足以,好吧,说实话是自己的电脑不行,使用虚拟机实在太卡了... 整个的集群搭建是在公司的测试服务搭建的,在搭建的时候遇到各种各样的坑,当然也收获颇多.在成功搭建大数据集群之后,零零散散的做了写笔记,然后重新将这些笔记整理了下来.于是就有了本篇博文. 其实我在搭

大数据学习系列之七 ----- Hadoop+Spark+Zookeeper+HBase+Hive集

引言 在之前的大数据学习系列中,搭建了Hadoop+Spark+HBase+Hive 环境以及一些测试.其实要说的话,我开始学习大数据的时候,搭建的就是集群,并不是单机模式和伪分布式.至于为什么先写单机的搭建,是因为作为个人学习的话,单机已足以,好吧,说实话是自己的电脑不行,使用虚拟机实在太卡了... 整个的集群搭建是在公司的测试服务搭建的,在搭建的时候遇到各种各样的坑,当然也收获颇多.在成功搭建大数据集群之后,零零散散的做了写笔记,然后重新将这些笔记整理了下来.于是就有了本篇博文. 其实我在搭

大数据学习路径

大数据学习路径我vx ①⑤零零③④③⑥⑨③① 备注 A 更多大数据第一阶段linux+搜索+hadoop体系 Linux基础→shell编程→高并发架构→lucene,solr搜索→hadoop体系→HDFS→mapreduce→hbase→zookeeper→hive→flume→sqoop→项目实战一 第二阶段机器学习 R语言→mahout→项目实战二 第三阶段storm流式计算 kafka→storm→redis→项目实战三 第四阶段spark内存计算 scala编程→spark core

Linux虚拟机学习环境搭建—虚拟机安装

接上一篇的Linux虚拟机学习环境搭建-虚拟机创建"  点击"开启此虚拟机",进入系统安装界面. Tab键可以调整选项,直接选择第一项,回车.备注:鼠标点击进入虚拟机,要跳出虚拟机回到主机控制,按下Ctrl+Alt. 出现Disc Found界面,用Tab键,选择到Skip选项,回车.备注:此操作是校验安装文件是否有损坏,现在是安装学习测试机,跳过校验. 进入到图标界面,单击 "下一步" 进入到语言选择界面,此处选择"English",

大数据学习系列之五 ----- Hive整合HBase图文详解

引言 在上一篇 大数据学习系列之四 ----- Hadoop+Hive环境搭建图文详解(单机) 和之前的大数据学习系列之二 ----- HBase环境搭建(单机) 中成功搭建了Hive和HBase的环境,并进行了相应的测试.本文主要讲的是如何将Hive和HBase进行整合. Hive和HBase的通信意图 Hive与HBase整合的实现是利用两者本身对外的API接口互相通信来完成的,其具体工作交由Hive的lib目录中的hive-hbase-handler-*.jar工具类来实现,通信原理如下图