hadoop0.20.2伪分布式环境搭建

虽然现在hadoop版本已经到了主流2点多的时代，但是对于学习大数据而言，我还是选择从以前老的版本0.20.2学起。

下面就是伪分布式的环境搭建过程。

hadoop下载地址：

http://archive.apache.org/dist/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz

linux系统版本：centos7

1、配置主机名

[[email protected] ~]# vi /etc/sysconfig/network

# Created by anaconda

master1

[[email protected] ~]# hostname master1

2、创建管理hadoop的组和用户

[[email protected] ~]# groupadd hduser

[[email protected] ~]# useradd -g hduser hduser

[[email protected] ~]# passwd hduser

3、hosts主机名ip解析

[[email protected] ~]# vi /etc/hosts

192.168.11.131  master1

4、配置hadoop的sudoers权限

[[email protected] ~]# vi /etc/sudoers

hduser  ALL=(ALL)       NOPASSWD:ALL

5、关闭selinux和防火墙

[[email protected] ~]# vi /etc/sysconfig/selinux

SELINUX=enforcing --> SELINUX=disabled

[[email protected] ~]# systemctl stop firewalld

[[email protected] ~]# systemctl disable firewalld

6、解压包

[[email protected] ~]# su hduser

[[email protected] root]$ cd

[[email protected] ~]$ ll *tar*

-rw-r--r--. 1 root root  44575568 Jun 16 17:24 hadoop-0.20.2.tar.gz

-rw-r--r--. 1 root root 288430080 Mar 16  2016 jdk1.7.0_79.tar

[[email protected] ~]$ tar xf jdk1.7.0_79.tar

[[email protected] ~]$ tar zxf hadoop-0.20.2.tar.gz

[[email protected] ~]$ mv jdk1.7.0_79 jdk

[[email protected] ~]$ mv hadoop-0.20.2 hadoop

7、配置java环境

[[email protected] ~]$ vi .bashrc

export JAVA_HOME=/home/hduser/jdk

export JRE_HOME=$JAVA_HOME/jre

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=./:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

[[email protected] ~]$ source .bashrc

[[email protected] ~]$ java -version

java version "1.7.0_79"

Java(TM) SE Runtime Environment (build 1.7.0_79-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

8、配置hadoop

[[email protected] conf]$ pwd

/home/hduser/hadoop/conf

[[email protected] conf]$ vi hadoop-env.sh

export JAVA_HOME=/home/hduser/jdk

[[email protected] conf]$ vi core-site.xml

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://master1:9000</value>

</property>

</configuration>

[[email protected] conf]$ sudo mkdir -p /data/hadoop/data

[[email protected] conf]$ sudo chown -R hduser:hduser /data/hadoop/data

[[email protected] conf]$ vi hdfs-site.xml

<configuration>

<property>

<name>dfs.data.dir</name>

<value>/data/hadoop/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

[[email protected] conf]$ vi mapred-site.xml

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>master1:9001</value>

</property>

</configuration>

9、做无密码认证

[[email protected] conf]$ cd

[[email protected] ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hduser/.ssh/id_rsa):

Created directory '/home/hduser/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hduser/.ssh/id_rsa.

Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:qRJhPSF32QDs9tU3e0/mAx/EBC2MHamGv2WPvUw19/M [email protected]

The key's randomart image is:

+---[RSA 2048]----+

|    ..+.o+  +o=  |

|     +.o. .. = o |

|    o.o   ... +  |

|   . .o. o.o. oo |

|    .. .S.o  ..+o|

|     . ..  . +..O|

|    . .     + *B+|

|     .     . .o==|

|               oE|

+----[SHA256]-----+

一路enter键

[[email protected] ~]$ cd .ssh

[[email protected] .ssh]$ ls

id_rsa  id_rsa.pub

[[email protected] .ssh]$ cp id_rsa.pub authorized_keys

10、格式化文件系统

[[email protected] .ssh]$ cd

[[email protected] ~]$ cd  hadoop/bin

[[email protected] bin]$ ./hadoop namenode -format

18/06/19 04:02:12 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = master1/192.168.11.131

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 0.20.2

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010

************************************************************/

18/06/19 04:02:13 INFO namenode.FSNamesystem: fsOwner=hduser,hduser

18/06/19 04:02:13 INFO namenode.FSNamesystem: supergroup=supergroup

18/06/19 04:02:13 INFO namenode.FSNamesystem: isPermissionEnabled=true

18/06/19 04:02:13 INFO common.Storage: Image file of size 96 saved in 0 seconds.

18/06/19 04:02:13 INFO common.Storage: Storage directory /tmp/hadoop-hduser/dfs/name has been successfully formatted.

18/06/19 04:02:13 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master1/192.168.11.131

************************************************************/

11、启动服务

[[email protected] bin]$ ./start-all.sh

starting namenode, logging to /home/hduser/hadoop/bin/../logs/hadoop-hduser-namenode-master1.out

The authenticity of host 'localhost (::1)' can't be established.

ECDSA key fingerprint is SHA256:OXYl4X6F6g4TV7YriZaSvuBIFM840h/qTg8/B7BUil0.

ECDSA key fingerprint is MD5:b6:b6:04:2d:49:70:8b:ed:65:00:e2:05:b0:95:5b:6d.

Are you sure you want to continue connecting (yes/no)? yes

localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.

localhost: starting datanode, logging to /home/hduser/hadoop/bin/../logs/hadoop-hduser-datanode-master1.out

localhost: starting secondarynamenode, logging to /home/hduser/hadoop/bin/../logs/hadoop-hduser-secondarynamenode-master1.out

starting jobtracker, logging to /home/hduser/hadoop/bin/../logs/hadoop-hduser-jobtracker-master1.out

localhost: starting tasktracker, logging to /home/hduser/hadoop/bin/../logs/hadoop-hduser-tasktracker-master1.out

12、查看服务

[[email protected] bin]$ jps

1867 JobTracker

1804 SecondaryNameNode

1597 NameNode

1971 TaskTracker

2011 Jps

1710 DataNode

[[email protected] bin]$

13、浏览器查看服务状态

使用web查看HSFS运行状态

在浏览器输入

http://192.168.11.131:50030

使用web查看MapReduce运行状态

在浏览器输入

http://192.168.11.131:50070

原文地址：http://blog.51cto.com/xiaoxiaozhou/2131518

时间： 2024-10-03 23:04:15

hadoop0.20.2伪分布式环境搭建的相关文章

hadoop-0.20.2伪分布式安装简记

1.准备环境 虚拟机(redhat enterprise linux 6.5) jdk-8u92-linux-x64.tar.gz hadoop-0.20.2.tar.gz 2.关闭虚拟机的防火墙,selinux,配置SSH免密码登录 [[email protected] ~]# vim /etc/sysconfig/selinux [[email protected] ~]# iptables -F [[email protected] ~]# service iptables save ip

【Hadoop】伪分布式环境搭建、验证

Hadoop伪分布式环境搭建: 自动部署脚本: #!/bin/bash set -eux export APP_PATH=/opt/applications export APP_NAME=Ares # 安装apt依赖包 apt-get update -y && apt-get install supervisor -y && apt-get install python-dev python-pip libmysqlclient-dev -y # 安装pip.python

一、Hadoop伪分布式环境搭建

Hadoop 2.x伪分布式环境搭建步骤: 1.修改hadoop-env.sh.yarn-env.sh.mapred-env.sh 方法:使用notepad++(beifeng用户)代开这三个文件 添加代码:export JAVA_HOME=/opt/modules/jdk1.7.0_67 2.修改core-site.xml.hdfs-site.xml.yarn-site.xml.mapred-site.xml配置文件 1)修改core-site.xml <configuration> <

Hadoop 2.x伪分布式环境搭建测试

Hadoop 2.x伪分布式环境搭建测试 标签(空格分隔): hadoop hadoop,spark,kafka交流群:459898801 1,搭建hadoop所需环境 卸载open JDK rpm -qa |grep java rpm -e –nodeps [java] 1.1,在/opt/目录下创建四个目录: modules/ software/ datas/ tools/ 解压hadoop-2.5.0及jdk-7u67-linux-x64.tar.gz至modules目录下. $tar -

linux环境下的伪分布式环境搭建

本文的配置环境是VMware10+centos2.5. 在学习大数据过程中,首先是要搭建环境,通过实验,在这里简短粘贴书写关于自己搭建大数据伪分布式环境的经验. 如果感觉有问题,欢迎咨询评论. 一:伪分布式准备工作 1.规划目录 2.修改目录所有者和所属组 3.删除原有的jdk 4.上传需要的jdk包 5.增加jdk 的执行权限 6.解压jdk 7.修改profile的JAVA_HOME,PATH 8.切换至root用户,使文件生效 9.检验jdk是否成功 二:搭建为分布式 1.解压hadoop

《OD大数据实战》hadoop伪分布式环境搭建

一.安装并配置Linux 8. 使用当前root用户创建文件夹,并给/opt/下的所有文件夹及文件赋予775权限,修改用户组为当前用户 mkdir -p /opt/modules mkdir -p /opt/software mkdir -p /opt/datas mkdir -p /opt/tools chmod 775 /opt/* chown beifeng:beifeng /opt/* 最终效果如下: [[email protected]02 opt]$ pwd /opt [[email

Hadoop2.x伪分布式环境搭建(一)

1.安装hadoop环境,以hadoop-2.5.0版本为例,搭建伪分布式环境,所需要工具包提供网盘下载:http://pan.baidu.com/s/1o8HR0Qu 2.上传所需要的工具包到linux相对就应的目录中 3.接上篇(Linux基础环境的各项配置(三)中最后一部分,需卸载系统自带的jdk,以免后续安装的jdk产生冲突),卸载jdk完成后,安装jdk-7u67-linux-x64.tar.gz版本,上述工具包可下载 (1).解压JDK tar -zxf jdk-7u67-linux

HDFS伪分布式环境搭建

(一).HDFS shell操作 以上已经介绍了如何搭建伪分布式的Hadoop,既然环境已经搭建起来了,那要怎么去操作呢?这就是本节将要介绍的内容: HDFS自带有一些shell命令,通过这些命令我们可以去操作HDFS文件系统,这些命令与Linux的命令挺相似的,如果熟悉Linux的命令很容易就可以上手HDFS的命令,关于这些命令的官方文档地址如下: http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/hadoop-mapre

Hadoop学习笔记之Hadoop伪分布式环境搭建

搭建为伪分布式Hadoop环境 1.宿主机(Windows)与客户机(安装在虚拟机中的Linux)网络连接. a) Host-only 宿主机与客户机单独组网: 好处:网络隔离: 坏处:虚拟机和其他服务器之间不能通讯: b) Bridge 桥接 宿主机与客户机在同一个局域网中. 好处:窦在同一个局域网,可以互相访问: 坏处:不完全. 2.Hadoop的为分布式安装步骤 a) 设置静态IP 在centos下左面上右上角图标右键修改: 重启网卡service network restart; 验证: