一、环境准备
①准备三台虚拟机,配置静态IP
②先修改主机名(每个节点统一命名规范)
vim /etc/hostname master #重启生效
配置DNS每个节点
vim /etc/hosts 192.168.60.121 master192.168.60.122 salve1 192.168.60.123 salve2
永久关闭防火墙
systemctl stop firewalld systemctl disable firewalld
配置免密登录
ssh-keygen -t rsa #一路回车即可cd 到 .sshcp id_rsa.pub authorized_keys #生成公钥
将公钥拷贝到节点
scp authorized_keys [email protected]:/root/.ssh/
scp authorized_keys [email protected]:/root/.ssh/
登录到hadoop2主机cd到.ssh
cat id_isa.pub >> authorized_keys #使用cat追加方式
登录到2号主机重复操作,再将公钥拷贝到三台主机上
二、配置jdk1.8
将jdk解压到自定义目录
vim /etc/profile #添加如下信息export JAVA_HOME=jdk安装目录export CLASSPATH=$JAVA_HOME/lib/export PATH=$PATH:JAVA_HOME/bin再保存执行 #source /etc/profile验证#java -version
三、Hadoop环境配置
解压并移动到自定义位置
vim /etc/profile export HADOOP_HOME=Hadoop的安装目录 export PATH=$PAHT:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/Hadoop更新资源使生效source /etc/profil
首先在hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中指定JDK的路径
export JAVA_HOME=jdk安装目录
配置core-site.xml
<configuration> <property> <name>fs.checkpoint.period</name> <value>3600</value> </property> <property> <name>fs.checkpoint.size</name> <value>67108864</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://node1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/data/hdfs/tmp</value> </property> <property> <name>hadoop.http.staticuser.user</name> <value>root</value> </property> </configuration>
配置hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/data/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/data/hdfs/data</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>node1:50090</value> </property> <property> <name>dfs.namenode.http-address</name> <value>node1:50070</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>file:/usr/local/data/hdfs/checkpoint</value> </property> <property> <name>dfs.namenode.checkpoint.edits.dir</name> <value>file:/usr/local/data/hdfs/edits</value> </property> </configuration>
配置yarn-site.xml
<property> <name>yarn.resourcemanager.hostname</name> <value>node1</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandle</value> </property> <property> <name>yarn.resourcemanager.resource-tarcker.address</name> <value>node1:8025</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>node1:8030</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>node1:8040</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>node1:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>node1:8088</value> </property>
配置mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapred.job.tarcker</name> <value>node1:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>node1:19888</value> </property> </configuration>
修改workers文件,删除localhost,并换成
slave1 slave2
新建文件夹(不建立应该也是可以的)
mkdir /usr/local/data/hdfs/tmp mkdir /usr/local/data/hdfs/name mkdir /usr/local/data/hdfs/data mkdir /usr/local/data/hdfs/checkpoint mkdir /usr/local/data/hdfs/edits
复制Hadoop文件到节点
scp -r /目的目录 hadoop2:./目的目录
Hadoop安装完成,格式化Namenode
cd到bin目录./Hdfs namenode -format
启动Hadoop
cd到sbin下 ./start-all.sh
OVER。。。
原文地址:https://www.cnblogs.com/jake-jin/p/11978376.html
时间: 2024-11-07 18:49:02