hadoop 伪分布式搭建

1.准备Linux环境
    1.0点击VMware快捷方式，右键打开文件所在位置 -> 双击vmnetcfg.exe -> VMnet1 host-only ->修改subnet ip 设置网段：192.168.8.0 子网掩码：255.255.255.0 -> apply -> ok
        回到windows --> 打开网络和共享中心 -> 更改适配器设置 -> 右键VMnet1 -> 属性 -> 双击IPv4 -> 设置windows的IP：192.168.8.100 子网掩码：255.255.255.0 -> 点击确定
        在虚拟软件上 --My Computer -> 选中虚拟机 -> 右键 -> settings -> network adapter -> host only -> ok    
    1.1修改主机名
        vim /etc/sysconfig/network
        
        NETWORKING=yes
        HOSTNAME=itcast01    ###

1.2修改IP
        两种方式：
        第一种：通过Linux图形界面进行修改（强烈推荐）
            进入Linux图形界面 -> 右键点击右上方的两个小电脑 -> 点击Edit connections -> 选中当前网络System eth0 -> 点击edit按钮 -> 选择IPv4 -> method选择为manual -> 点击add按钮 -> 添加IP：192.168.8.118 子网掩码：255.255.255.0 网关：192.168.1.1 -> apply
    
        第二种：修改配置文件方式（屌丝程序猿专用）
            vim /etc/sysconfig/network-scripts/ifcfg-eth0
            
            DEVICE="eth0"
            BOOTPROTO="static"               ###
            HWADDR="00:0C:29:3C:BF:E7"
            IPV6INIT="yes"
            NM_CONTROLLED="yes"
            ONBOOT="yes"
            TYPE="Ethernet"
            UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"
            IPADDR="192.168.8.118"           ###
            NETMASK="255.255.255.0"          ###
            GATEWAY="192.168.8.1"            ###
            
    1.3修改主机名和IP的映射关系
        vim /etc/hosts
            
        192.168.8.118    itcast01
    
    1.4关闭防火墙
        #查看防火墙状态
        service iptables status
        #关闭防火墙
        service iptables stop
        #查看防火墙开机启动状态
        chkconfig iptables --list
        #关闭防火墙开机启动
        chkconfig iptables off
    
    1.5重启Linux
        reboot

2.安装JDK
    2.1上传
    
    2.2解压jdk
        #创建文件夹
        mkdir /usr/java
        #解压
        tar -zxvf jdk-7u55-linux-i586.tar.gz -C /usr/java/
        
    2.3将java添加到环境变量中
        vim /etc/profile
        #在文件最后添加
        export JAVA_HOME=/usr/java/jdk1.7.0_55
        export PATH=$PATH:$JAVA_HOME/bin
    
        #刷新配置
        source /etc/profil
        
3.安装hadoop2.4.1
    注意：hadoop2.x的配置文件$HADOOP_HOME/etc/hadoop
    伪分布式需要修改5个配置文件
    3.1配置hadoop
    第一个：hadoop-env.sh
        vim hadoop-env.sh
        #第27行
        export JAVA_HOME=/usr/java/jdk1.7.0_65
        
    第二个：core-site.xml
        <!-- 制定HDFS的老大（NameNode）的地址 -->
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://itcast01:9000</value>
        </property>
        <!-- 指定hadoop运行时产生文件的存储目录 -->
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/itcast/hadoop-2.4.1/tmp</value>
        </property>
        
    第三个：hdfs-site.xml
        <!-- 指定HDFS副本的数量 -->
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
        
    第四个：mapred-site.xml (mv mapred-site.xml.template mapred-site.xml)
        mv mapred-site.xml.template mapred-site.xml
        vim mapred-site.xml
        <!-- 指定mr运行在yarn上 -->
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
        
    第五个：yarn-site.xml
        <!-- 指定YARN的老大（ResourceManager）的地址 -->
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>itcast01</value>
        </property>
        <!-- reducer获取数据的方式 -->
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    
    3.2将hadoop添加到环境变量
    
    vim /etc/proflie
        export JAVA_HOME=/usr/java/jdk1.7.0_65
        export HADOOP_HOME=/itcast/hadoop-2.4.1
        export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile
    
    3.3格式化namenode（是对namenode进行初始化）
        hdfs namenode -format (hadoop namenode -format)
        
    3.4启动hadoop
        先启动HDFS
        sbin/start-dfs.sh
        
        再启动YARN
        sbin/start-yarn.sh
        
    3.5验证是否启动成功
        使用jps命令验证
        27408 NameNode
        28218 Jps
        27643 SecondaryNameNode
        28066 NodeManager
        27803 ResourceManager
        27512 DataNode
    
        http://192.168.8.118:50070 （HDFS管理界面）
        http://192.168.8.118:8088 （MR管理界面）
        
4.配置ssh免登陆
    #生成ssh免登陆密钥
    #进入到我的home目录
    cd ~/.ssh

ssh-keygen -t rsa （四个回车）
    执行完这个命令后，会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）
    将公钥拷贝到要免登陆的机器上
    ssh-copy-id localhost

开启Hadoop时，出现如下信息：

[[email protected]1 /]# ./hadoop/hadoop-2.6.0/sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/01/23 20:23:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [Java HotSpot(TM) Client VM warning: You have loaded library /hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
hd-m1]
sed: -e expression #1, char 6: unknown option to `s‘
-c: Unknown cipher type ‘cd‘
hd-m1: starting namenode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-namenode-hd-m1.out
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
Client: ssh: Could not resolve hostname Client: Temporary failure in name resolution
You: ssh: Could not resolve hostname You: Temporary failure in name resolution
warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
might: ssh: Could not resolve hostname might: Temporary failure in name resolution
which: ssh: Could not resolve hostname which: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
The: ssh: Could not resolve hostname The: Temporary failure in name resolution
try: ssh: Could not resolve hostname try: Temporary failure in name resolution
will: ssh: Could not resolve hostname will: Temporary failure in name resolution
to: ssh: Could not resolve hostname to: Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
It‘s: ssh: Could not resolve hostname It‘s: Temporary failure in name resolution
now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
that: ssh: Could not resolve hostname that: Temporary failure in name resolution
you: ssh: Could not resolve hostname you: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
< libfile>‘,: ssh: Could not resolve hostname <libfile>‘,: Temporary failure in name resolution
or: ssh: Could not resolve hostname or: Temporary failure in name resolution
link: ssh: Could not resolve hostname link: Temporary failure in name resolution
it: ssh: Could not resolve hostname it: Temporary failure in name resolution
‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
noexecstack‘.: ssh: Could not resolve hostname noexecstack‘.: Temporary failure in name resolution
hd-s1: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-hd-s1.out
hd-s2: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-hd-s2.out
Starting secondary namenodes [Java HotSpot(TM) Client VM warning: You have loaded library /hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
SecondaryNameNode]
sed: -e expression #1, char 6: unknown option to `s‘
-c: Unknown cipher type ‘cd‘
Client: ssh: Could not resolve hostname Client: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
You: ssh: Could not resolve hostname You: Temporary failure in name resolution
Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
might: ssh: Could not resolve hostname might: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
to: ssh: Could not resolve hostname to: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
It‘s: ssh: Could not resolve hostname It‘s: Temporary failure in name resolution
disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
that: ssh: Could not resolve hostname that: Temporary failure in name resolution
recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
try: ssh: Could not resolve hostname try: Temporary failure in name resolution
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
‘execstack: ssh: Could not resolve hostname ‘execstack: Temporary failure in name resolution
warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
or: ssh: Could not resolve hostname or: Temporary failure in name resolution
< libfile>‘,: ssh: Could not resolve hostname <libfile>‘,: Temporary failure in name resolution
you: ssh: Could not resolve hostname you: Temporary failure in name resolution
link: ssh: Could not resolve hostname link: Temporary failure in name resolution
it: ssh: Could not resolve hostname it: Temporary failure in name resolution
which: ssh: Could not resolve hostname which: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
The: ssh: Could not resolve hostname The: Temporary failure in name resolution
noexecstack‘.: ssh: Could not resolve hostname noexecstack‘.: Temporary failure in name resolution
‘-z: ssh: Could not resolve hostname ‘-z: Temporary failure in name resolution
will: ssh: Could not resolve hostname will: Temporary failure in name resolution
SecondaryNameNode: ssh: Could not resolve hostname SecondaryNameNode: Temporary failure in name resolution
guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
15/01/23 20:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-root-resourcemanager-hd-m1.out
hd-s1: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-hd-s1.out
hd-s2: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-hd-s2.out

解决办法：

出现上述问题主要是环境变量没设置好，在~/.bash_profile或者/etc/profile中加入以下语句就没问题了。

　　#vi /etc/profile或者vi ~/.bash_profile
　　　　export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
　　　　export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

然后用source重新编译使之生效即可！
　　#source /etc/profile或者source ~/.bash_profile

jdk 下载地址为：

http://pan.baidu.com/s/1o6ED0IQ

hadoop2.4.1 64位下载地址为:

http://pan.baidu.com/s/1bnlnJyv

时间： 2024-11-06 07:39:27

hadoop 伪分布式搭建的相关文章

Hadoop - Hadoop伪分布式搭建

进行Hadoop伪分布式搭建时,需要有一定的Linux命令基础, 因为Hadoop是搭建在Linux环境上的开源框架, Hadoop的框架最核心的设计就是:HDFS和MapReduce. 本文也主要讲述HDFS和MapReduce环境的搭建. 搭建环境: 虚拟机: VMware10 操作系统:CentOS-6.5 JDK版本:jdk-8u171-linux-x64 Hadoop版本:hadoop-2.7.3 使用工具:Xshell 6,Xftp 6,Notepad++ 虚拟机ip地址:192.1

Hadoop伪分布式搭建步骤

说明: 搭建环境是VMware10下用的是Linux CENTOS 32位,Hadoop:hadoop-2.4.1  JAVA :jdk7 32位:本文是本人在网络上收集的HADOOP系列视频所附带的课堂笔记(这里直接抄过来的,望原谅,我不是坏人),记录备忘. 开始前让我们简单修改下LINUX系统的JDK:一般情况下,我们都要将linux自带的OPENJDK卸载掉,然后安装SUN的JDK(下面的代码来源网络,请注意真实自己环境) <1> 查看已经安装的JAVA版本信息 # java -vers

hadoop伪分布式搭建

1.搭建伪分布式集群使用root用户登录 第一步:设置ip,为虚拟机设置一个ip地址: 设置完成后要检测:service network restart 重启生效,然后ping命令测试该ip是不是能ping通. 2.更改主机名: 执行命令:vi  /etc/sysconfig/network 更改之后再重新启动就会更改用户名,通过命令hostname即可查看 3.设置hosts中ip与主机名的绑定 执行命令 vi  /etc/hosts 在里面添加选项设置你的主机名与ip的对应关系 Ping一下

Hadoop伪分布式搭建CentOS

所需软件及版本: jdk-7u80-linux-x64.tar.gz hadoop-2.6.0.tar.gz 1.安装JDK Hadoop 在需在JDK下运行,注意JDK最好使用Oracle的否则可能出bug 2.创建用户 [[email protected] ~]# groupadd hadoop [[email protected] ~]# useradd -g hadoop hadoop [[email protected] ~]# passwd hadoop 3.配置SSH 确认SSH的

虚拟机hadoop伪分布式搭建

虚拟机搭建自定义类型 主机名为hadoop可以省略 很多步骤 1.安装虚拟机 配置和网络 (1)IPADDR (2)NETMASK (3)GATEWAY (4)DNS1 远程连接 Xshell 关闭防火墙 (1)service iptables stop (2)chkconfig iptables off 2.安装jdk,配置环境变量 上传jdk压缩包 解压 jdk放到/usr/local路径下 (1)tar -zxvf  解压 (2)ln -s xxxx.jdk jdk 建立软连接 配置环境变

hadoop2.2.0伪分布式搭建

一.准备linux环境 1.更改VMware适配器设置 由于是在单机环境下进行学习的,因此选择适配器模式是host-only模式,如果想要联网,可以选择桥接模式,配置的方式差不多. 点击VMware快捷方式,右键打开文件所在位置 -> 双击vmnetcfg.exe -> VMnet1 host-only ->修改subnet ip 设置网段:192.168.85.0 子网掩码:255.255.255.0 -> apply -> ok 回到windows --> 打开网络

Hadoop初体验：快速搭建Hadoop伪分布式环境

0.前言 本文旨在使用一个全新安装好的Linux系统从0开始进行Hadoop伪分布式环境的搭建,以达到快速搭建的目的,从而体验Hadoop的魅力所在,为后面的继续学习提供基础环境. 对使用的系统环境作如下说明: 操作系统:CentOS 6.5 64位 主机IP地址:10.0.0.131/24 主机名:leaf 用户名:root hadoop版本:2.6.5 jdk版本:1.7 可以看到,这里直接使用root用户,而不是按照大多数的教程创建一个hadoop用户来进行操作,就是为了达到快速搭建Had

Hadoop简单入门之伪分布式搭建

前面两章主要讲解了完全分布式的搭建,这章主要讲解服务器单机完成伪分布的搭建,介绍Hadoop配置,启动,以及简单测试.我的机器:阿里云服务器,64位,Java64,Hadoop2.4.1(64) ? 所有软件下载百度云 密码:uup8 讲在开头:对于笔者的完全分布式环境请见该文:Hadoop完全分布式安装 写文章不易,欢迎大家采我的文章,以及给出有用的评论,当然大家也可以关注一下我的github:多谢: 1,Hadoop简单介绍: Apache Hadoop是一款支持数据密集型分布式应用,并以A

Hadoop伪分布式集群搭建总结

Hadoop伪分布式集群搭建总结 一.所需软件VMware15!CentOS6.5JDK1.8Hadoop2.7.3二.安装注意:对文件进行编辑:输入a,表示对该文件进行编辑,最后保存该文件,操作为:点击键盘上的Esc按钮,然后输入英文的:字符,再输入wq,点击回车,完成文件的保存.1.关闭防火墙和禁用SELINUX(1).永久关闭防火墙,重启Linux系统(2) .禁用SELINUX:修改文件参数 重启Linux使其生效(3).检查防火墙是否运行,显示下图即为关闭2.配置hostname与IP