1、软件环境
RHEL6 jdk-8u45
hadoop-2.8.1.tar.gz ssh
xx.xx.xx.xx ip地址 hadoop1
xx.xx.xx.xx ip地址 hadoop2
xx.xx.xx.xx ip地址 hadoop3
xx.xx.xx.xx ip地址 hadoop4
xx.xx.xx.xx ip地址 hadoop5
本次部署只涉及伪分布式部署只是要主机hadoop1
2.伪分布式部署
伪分布式部署文档参考官方网站hadoop
Apache > Hadoop > Apache Hadoop Project Dist POM > Apache Hadoop 2.8.3 > Hadoop: Setting up a Single Node Cluster
添加hadoop用户
[[email protected] ~]# useradd hadoop
[[email protected] ~]# vi /etc/sudoers
找到root ALL=(ALL) ALL，添加
hadoop ALL=(ALL) NOPASSWD:ALL
上传并解压
[[email protected] software]# rz #上传hadoop-2.8.1.tar.gz
[[email protected] software]# tar -xzvf hadoop-2.8.1.tar.gz
软连接
[[email protected] software]# ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop
设置环境变量
[[email protected] software]# vi /etc/profile
export HADOOP_HOME=/opt/software/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
[[email protected] software]# source /etc/profile
设置用户、用户组
[[email protected] software]# chown -R hadoop:hadoop hadoop
[[email protected] software]# chown -R hadoop:hadoop hadoop/
[[email protected] software]# chown -R hadoop:hadoop hadoop-2.8.1
[[email protected] software]# cd hadoop
[[email protected] hadoop]# rm -f .txt
[[email protected] software]# su - hadoop
[[email protected] hadoop]# ll
total 32
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 bin
drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 etc
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 include
drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 lib
drwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13:59 libexec
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13:59 logs
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 sbin
drwxrwxr-x. 4 hadoop hadoop 4096 Jun 2 14:24 share*
bin: 可执行文件
etc: 配置文件
sbin: shell脚本，启动关闭hdfs,yarn等配置文件
[[email protected] ~]# cd /opt/software/hadoop
[[email protected] hadoop]# vi etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://172.16.18.133:9000</value> # 配置自己机器的IP
</property>
</configuration>
[[email protected] hadoop]# vi etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
配置hadoop用户的ssh信任关系
公钥/密钥 配置无密码登录
[[email protected] ~]# ssh-keygen -t rsa -P ‘‘ -f ~/.ssh/id_rsa
[[email protected] ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[[email protected] ~]# chmod 0600 ~/.ssh/authorized_keys
[[email protected] ~]# ssh hadoop1 date
Wed Apr 25 15:22:58 CST 2018
[[email protected] ~]# ssh localhost date
Wed Apr 25 15:23:25 CST 2018
第一次执行命令有交互选择输入yes
格式化和启动
[[email protected] hadoop]# bin/hdfs namenode -format
[[email protected] hadoop]# sbin/start-dfs.sh
ERROR:
hadoop1: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
解决方法:添加环境变量
[[email protected] hadoop]# vi etc/hadoop/hadoop-env.sh
将export JAVA_HOME=${JAVA_HOME}改为
export JAVA_HOME=/usr/java/jdk1.8.0_45
继续启动
[[email protected] hadoop]# sbin/start-dfs.sh
检查是否成功
[[email protected] hadoop]# jps
8026 Jps
7485 NameNode
7597 DataNode
7807 SecondaryNameNode
访问： http://192.168.137.130:50070
修改dfs启动的进程，以hadoop1启动 启动的三个进程：
namenode: hadoop1 bin/hdfs getconf -namenodes
datanode: localhost datanodes (using default slaves file) etc/hadoop/slaves
secondarynamenode: 0.0.0.0
[[email protected] ~]# cd /opt/software/hadoop
[[email protected] hadoop]# echo "hadoop1" > ./etc/hadoop/slaves
[[email protected] hadoop]# cat ./etc/hadoop/slaves
hadoop1
[[email protected] hadoop]# vi ./etc/hadoop/hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop1:50090</value>
</property>
<property>
<name>dfs.namenode.secondary.https-address</name>
<value>hadoop1:50091</value>
</property>
[[email protected] hadoop]# sbin/stop-dfs.sh
[[email protected] hadoop]# sbin/start-dfs.sh
访问： http://172.16.18.133:50070界面
原文地址：http://blog.51cto.com/chaorenyong/2116627