VMware下Hadoop 2.4.1完全分布式集群平台安装与设置

1 VM下Ubuntu安装和配置

1.1 安装Ubuntu系统

这个就不说了，不知道的可以去看看其他的博文。

1.2 集群配置

搭建一个由3台机器组成的集群：

IP user/passwd hostname role System
192.168.174.160 hadoop/hadoop master nn/snn/rm Ubuntu-14.04-32bit
192.168.174.161 hadoop/hadoop slave1 dn/nm Ubuntu-14.04-32bit
192.168.174.162 hadoop/hadoop slave2 dn/nm Ubuntu-14.04-32bit

nn：    namenode            snn：  secondary namenode

dn：    datanode            rm：    resourcemanager

nm：    nodemanager

1.3 创建用户
    我这里给每台机器创建一个hadoop的用户，密码也为hadoop，修改/etc/sudoers文件，增加：hadoop  ALL=(ALL) ALL，给每个账户分配sudo的权限。

[email protected]:/home/duanwf# useradd --create-home hadoop

[email protected]:/home/duanwf# passwd hadoop
 
[email protected]:~# vi /etc/sudoers
# User privilege specification
root ALL=(ALL:ALL) ALL
duanwf ALL=(ALL:ALL) ALL
hadoop ALL=(ALL:ALL) ALL


1.4 设定电脑的IP为静态地址

参考博文“Vmware下Ubuntu 14.04静态IP地址的设置方法”。

1.5 设置各个主机的hostname

　　打开/etc/hostname文件：

[email protected]:~# vi /etc/hostname 
master

1.6 在以上三台电脑的/etc/hosts添加以上配置的hostname

[email protected]:~# vi /etc/hosts 
127.0.0.1 localhost 
127.0.1.1 ubuntu 

# The following lines are desirable for IPv6 capable hosts 
::1 ip6-localhost ip6-loopback 
fe00::0 ip6-localnet 
ff00::0 ip6-mcastprefix 
ff02::1 ip6-allnodes 
ff02::2 ip6-allrouters 

192.168.174.160 master
192.168.174.161 slave1
192.168.174.162 slave2

1.7 设置SSH无密码登陆

安装SSH

[email protected]:~$ sudo apt-get install ssh

查看SSH是否安装成功以及版本

[email protected]:~$ ssh -V 
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014

安装完成后会在~目录（当前用户主目录，即这里的/home/hadoop）下产生一个隐藏文件夹.ssh（ls  -a 可以查看隐藏文件）。如果没有这个文件，自己新建即可（mkdir .ssh）。

[email protected]:~$ cd /home/hadoop 

[email protected]:~$ ls -a

[email protected]:~$ mkdir .ssh

进入.ssh文件夹

[email protected]:~$ cd .ssh

产生秘钥

[email protected]:~/.ssh$ ssh-keygen -t rsa 
Generating public/private rsa key pair. 
Enter file in which to save the key (/home/duanwf/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/duanwf/.ssh/id_rsa. 
Your public key has been saved in /home/duanwf/.ssh/id_rsa.pub. 
The key fingerprint is: 
49:ad:12:42:36:15:c8:f6:42:08:c1:d9:a6:04:27:a1 [email protected] 
The key‘s randomart image is: 
+--[ RSA 2048]----+ 
|O++o+oo. | 
|.*.==. . | 
|E oo... . . | 
| . ...o o | 
| .. S | 
| . | 
| | 
| | 
| | 
+-----------------+

把id_rsa.pub 追加到授权的 key 里面去

[email protected]:~/.ssh$ cat id_rsa.pub >> authorized_keys

重启 SSH 服务命令使其生效

[email protected]:~/.ssh$ service ssh restart 
stop: Rejected send message, 1 matched rules; type="method_call", sender=":1.109" (uid=1000 pid=8874 comm="stop ssh ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init") 
start: Rejected send message, 1 matched rules; type="method_call", sender=":1.110" (uid=1000 pid=8868 comm="start ssh ") interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")

将master的authorized_keys追加到slave1和slave2的authorized_keys：

把所有主机都配好之后再来完成该步骤。

这里只有一个是master，如果有多个namenode，或者rm的话则需要打通所有master都其他剩余节点的免密码登陆。

[email protected]:~/.ssh$ scp authorized_keys [email protected]:~/.ssh/authorized_keys_from_master 
The authenticity of host ‘slave1 (192.168.174.161)‘ can‘t be established. 
ECDSA key fingerprint is 1f:c0:2a:ed:c1:7b:6e:26:46:e3:c3:b6:87:bb:99:42. 
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added ‘slave1,192.168.174.161‘ (ECDSA) to the list of known hosts. 
[email protected]‘s password: 
authorized_keys 100% 395 0.4KB/s 00:00
 
[email protected]:~/.ssh$ scp authorized_keys [email protected]:~/.ssh/authorized_keys_from_master 
The authenticity of host ‘slave2 (192.168.174.162)‘ can‘t be established. 
ECDSA key fingerprint is 1f:c0:2a:ed:c1:7b:6e:26:46:e3:c3:b6:87:bb:99:42. 
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added ‘slave2,192.168.174.162‘ (ECDSA) to the list of known hosts. 
[email protected]‘s password: 
authorized_keys 100% 395 0.4KB/s 00:00

进入slave1和slave2的.ssh目录

[email protected]:~$ cd .ssh
[email protected]:~/.ssh$ ssh -V 
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014
[email protected]:~/.ssh$ cat authorized_keys_from_master >> authorized_keys 
[email protected]:~/.ssh$ ls 
authorized_keys authorized_keys_from_master
 
[email protected]:~/.ssh$ ssh -V 
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014 
[email protected]:~/.ssh$ cat authorized_keys_from_master >> authorized_keys 
[email protected]:~/.ssh$ ls 
authorized_keys authorized_keys_from_master

验证SSH无密码登录

[email protected]:~/.ssh$ ssh slave1 
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic i686) 

* Documentation: https://help.ubuntu.com/ 
packages can be updated. 
updates are security updates. 

Last login: Tue Oct 7 18:25:31 2014 from 192.168.174.1

如上显示，说明已经安装成功，第一次登录时会询问你是否继续链接，输入yes即可以进入。

　　实际上，在hadoop的安装过程中，是否无密码登陆不是必须的，但是如果不配置无密码登陆的话，每次启动hadoop，都需要输入密码以登陆到每台daotanode，考虑到一般的hadoop集群动辄数十数百台机器，因此一般来说都会配置ssh的无密码登陆。

2 JDK安装和配置

2.1 JDK下载

2.2 解压下载的jdk到/opt目录下

[email protected]:/home/duanwf/Installpackage$ sudo tar zxvf jdk-7u51-linux-i586.tar.gz -C /opt/

2.3 jdk环境变量配置

在终端输入以下命令来修改文件/etc/profile

[email protected]:~$ sudo vi /etc/profile

在文件最后加上

export JAVA_HOME=/opt/jdk1.7.0_51 
export JRE_HOME=/opt/jdk1.7.0_51/jre 
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH 
export PATH=$JAVA_HOME/bin:$PATH

使修改生效，执行命令：　　

[email protected]:~$ source /etc/profile

验证java环境变量是否配置成功：

[email protected]:~$ java -version 
java version "1.7.0_51" Java(TM) SE Runtime Environment (build 1.7.0_51-b13) 
Java HotSpot(TM) Client VM (build 24.51-b03, mixed mode)

3、防火墙配置

ubuntu下关闭防火墙，执行命令：

[email protected]:~$ sudo ufw disable

4、Hadoop-2.4.1的安装和配置

4.1 编译hadoop-2.4.1-src.tar.gz源包

对于64位操作系统，需要重新编译源码包。

Hadoop下载地址：http://mirrors.hust.edu.cn/apache/hadoop/common/

4.2 解压安装包hadoop-2.4.1.tar.gz

[email protected]:/home/duanwf/Installpackage$ sudo tar zxvf hadoop-2.4.1.tar.gz -C /opt/

4.3 Hadoop环境变量配置

修改/etc/profile文件，加入以下内容：

[email protected]:~$ sudo vi /etc/profile
export HADOOP_DEV_HOME=/home/hadoop/hadoop-2.4.1/
export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME}
export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}
export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}
export YARN_HOME=${HADOOP_DEV_HOME}
export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export PATH=$HADOOP_DEV_HOME/bin:$HADOOP_DEV_HOME/sbin:$PATH

使修改的配置生效，在终端输入命令：

[email protected]:~$ source /etc/profile

查看Hadoop环境变量是否生效，在终端执行命令：

[email protected]:~$ hadoop 
Usage: hadoop [--config confdir] COMMAND 
where COMMAND is one of: 
fs run a generic filesystem user client 
version print the version 
jar <jar> run a jar file 
checknative [-a|-h] check native hadoop and compression libraries availability 
distcp <srcurl> <desturl> copy file or directories recursively 
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive 
classpath prints the class path needed to get the 
Hadoop jar and the required libraries 
daemonlog get/set the log level for each daemon 
or 
CLASSNAME run the class named CLASSNAME 

Most commands print help when invoked w/o parameters.

4.4 hadoop配置

配置之前，需要在master本地文件系统创建以下文件夹：

~/dfs/name

~/dfs/data

~/temp

[email protected]:~$ mkdir ~/dfs 

[email protected]:~$ mkdir ~/temp 

[email protected]:~$ mkdir ~/dfs/name 

[email protected]:~$ mkdir ~/dfs/data

这里要涉及到的配置文件有7个：

~/hadoop-2.4.1/etc/hadoop/hadoop-env.sh

~/hadoop-2.4.1/etc/hadoop/yarn-env.sh

~/hadoop-2.4.1/etc/hadoop/slaves

~/hadoop-2.4.1/etc/hadoop/core-site.xml

~/hadoop-2.4.1/etc/hadoop/hdfs-site.xml

~/hadoop-2.4.1/etc/hadoop/mapred-site.xml.template

~/hadoop-2.4.1/etc/hadoop/yarn-site.xml

<---------------------------------- hadoop-env.sh --------------------------------->
[email protected]:/opt/hadoop-2.4.1/etc/hadoop$ sudo vi hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/opt/jdk1.7.0_51/

<---------------------------------- yarn-env.sh --------------------------------->
[email protected]:/opt/hadoop-2.4.1/etc/hadoop$ sudo vi yarn-env.sh
# some Java parameters
export JAVA_HOME=/opt/jdk1.7.0_51/

<---------------------------------- slaves --------------------------------->
[email protected]:/opt/hadoop-2.4.1/etc/hadoop$ sudo vi slaves 
slave1 
slave2

<---------------------------------- core-site.xml --------------------------------->
[email protected]:~/hadoop-2.4.1/etc/hadoop$ sudo vi core-site.xml
<configuration>
       <property>
                 <name>fs.defaultFS</name>
                 <value>hdfs://master:9000</value>
       </property>
       <property>
                 <name>io.file.buffer.size</name>
                 <value>131072</value>
       </property>
       <property>
                 <name>hadoop.tmp.dir</name>
                 <value>/home/hadoop/temp/</value>
                 <description>Abase for other temporary directories.</description>
       </property>
       <property>
               <name>hadoop.proxyuser.hduser.hosts</name>
               <value>*</value>
       </property>
       <property>
                        <name>hadoop.proxyuser.hduser.groups</name>
               <value>*</value>
       </property>
</configuration>

<---------------------------------- hdfs-site.xml --------------------------------->
[email protected]:/opt/hadoop-2.4.1/etc/hadoop$ sudo vi hdfs-site.xml
<configuration>
 <property>
         <name>dfs.namenode.secondary.http-address</name>
         <value>master:9001</value>
        </property>
        <property>
         <name>dfs.namenode.name.dir</name>
         <value>/home/hadoop/dfs/name/</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/home/hadoop/dfs/data/</value>
        </property>
        <property>
         <name>dfs.replication</name>
         <value>3</value>
        </property>
        <property>
         <name>dfs.webhdfs.enabled</name>
                <value>true</value>
         </property>
</configuration>

<---------------------------------- mapred-site.xml.template --------------------------------->
[email protected]:/opt/hadoop-2.4.1/etc/hadoop$ sudo vi mapred-site.xml.template
<configuration>
 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
 </property>
 <property>
  <name>mapreduce.jobhistory.address</name>
   <value>master:10020</value>
 </property>
 <property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>master:19888</value>
       </property>
</configuration>

<---------------------------------- yarn-site.xml --------------------------------->
[email protected]:/opt/hadoop-2.4.1/etc/hadoop$ sudo vi yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
  <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
 <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>master:8032</value>
       </property>
 <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value> master:8030</value>
 </property>
 <property>
  <name>yarn.resourcemanager.resource-tracker.address</name>
   <value> master:8031</value>
  </property>
 <property>
   <name>yarn.resourcemanager.admin.address</name>
    <value> master:8033</value>
 </property>
  <property>
  <name>yarn.resourcemanager.webapp.address</name>
  <value> master:8088</value>
        </property>
</configuration>

4.5 复制到其他节点

进入slave1:

[email protected]:~$ scp -r [email protected]:/home/hadoop/hadoop-2.4.1/ /home/hadoop/

进入slave2:

[email protected]:~$ scp -r [email protected]:/home/hadoop/hadoop-2.4.1/ /home/hadoop/

4.6 Hadoop启动

(1)格式化HDFS

[email protected]:~/hadoop-2.4.1$ ./bin/hdfs namenode -format
14/10/08 18:43:05 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************ 
STARTUP_MSG: Starting NameNode 
STARTUP_MSG: host = master/192.168.174.160 
STARTUP_MSG: args = [-format] 
STARTUP_MSG: version = 2.4.1 
STARTUP_MSG: classpath = /home/hadoop/hadoop-2.4.1//etc/hadoop:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/hadoop-annotations-2.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/junit-4.8.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/zookeeper-3.4.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/hadoop-auth-2.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/hadoop-nfs-2.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/hadoop-common-2.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/common/hadoop-common-2.4.1-tests.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/hadoop-hdfs-2.4.1-tests.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/hadoop-hdfs-nfs-2.4.1.jar:/home/hadoop/hadoop-2.4.1//share/hadoop/hdfs/hadoop-hdfs-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-server-common-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/hadoop-annotations-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/junit-4.10.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.4.1.jar:/home/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/contrib/capacity-scheduler/*.jar 
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common -r 1604318; compiled by ‘jenkins‘ on 2014-06-21T05:43Z 
STARTUP_MSG: java = 1.7.0_51 
************************************************************/ 
14/10/08 18:43:05 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 
14/10/08 18:43:05 INFO namenode.NameNode: createNameNode [-format] 
14/10/08 18:43:06 WARN common.Util: Path /home/hadoop/dfs/name/ should be specified as a URI in configuration files. Please update hdfs configuration. 
14/10/08 18:43:06 WARN common.Util: Path /home/hadoop/dfs/name/ should be specified as a URI in configuration files. Please update hdfs configuration. 
Formatting using clusterid: CID-f1441872-89ef-4733-98df-454c18da5043 
14/10/08 18:43:06 INFO namenode.FSNamesystem: fsLock is fair:true 
14/10/08 18:43:06 INFO namenode.HostFileManager: read includes: 
HostSet( 
) 
14/10/08 18:43:06 INFO namenode.HostFileManager: read excludes: 
HostSet( 
) 
14/10/08 18:43:06 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 
14/10/08 18:43:06 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 
14/10/08 18:43:06 INFO util.GSet: Computing capacity for map BlocksMap 
14/10/08 18:43:06 INFO util.GSet: VM type = 32-bit 
14/10/08 18:43:06 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 
14/10/08 18:43:06 INFO util.GSet: capacity = 2^22 = 4194304 entries 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: defaultReplication = 3 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: maxReplication = 512 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: minReplication = 1 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: encryptDataTransfer = false 
14/10/08 18:43:06 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 
14/10/08 18:43:06 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE) 
14/10/08 18:43:06 INFO namenode.FSNamesystem: supergroup = supergroup 
14/10/08 18:43:06 INFO namenode.FSNamesystem: isPermissionEnabled = true 
14/10/08 18:43:06 INFO namenode.FSNamesystem: HA Enabled: false 
14/10/08 18:43:06 INFO namenode.FSNamesystem: Append Enabled: true 
14/10/08 18:43:06 INFO util.GSet: Computing capacity for map INodeMap 
14/10/08 18:43:06 INFO util.GSet: VM type = 32-bit 
14/10/08 18:43:06 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 
14/10/08 18:43:06 INFO util.GSet: capacity = 2^21 = 2097152 entries 
14/10/08 18:43:06 INFO namenode.NameNode: Caching file names occuring more than 10 times 
14/10/08 18:43:06 INFO util.GSet: Computing capacity for map cachedBlocks 
14/10/08 18:43:06 INFO util.GSet: VM type = 32-bit 
14/10/08 18:43:06 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 
14/10/08 18:43:06 INFO util.GSet: capacity = 2^19 = 524288 entries 
14/10/08 18:43:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 
14/10/08 18:43:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 
14/10/08 18:43:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 
14/10/08 18:43:06 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 
14/10/08 18:43:06 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 
14/10/08 18:43:06 INFO util.GSet: Computing capacity for map NameNodeRetryCache 
14/10/08 18:43:06 INFO util.GSet: VM type = 32-bit 
14/10/08 18:43:06 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 
14/10/08 18:43:06 INFO util.GSet: capacity = 2^16 = 65536 entries 
14/10/08 18:43:06 INFO namenode.AclConfigFlag: ACLs enabled? false 
Re-format filesystem in Storage Directory /home/hadoop/dfs/name ? (Y or N) Y 
14/10/08 18:43:10 INFO namenode.FSImage: Allocated new BlockPoolId: BP-215877782-192.168.174.160-1412764990823 
14/10/08 18:43:10 INFO common.Storage: Storage directory /home/hadoop/dfs/name has been successfully formatted. 
14/10/08 18:43:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 
14/10/08 18:43:11 INFO util.ExitUtil: Exiting with status 0 
14/10/08 18:43:11 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************ 
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.174.160 
************************************************************/

格式化HDFS：[email protected]:~/hadoop-2.4.1$ ./bin/hdfs namenode -format

(2)启动HDFS

执行一下命令启动HDFS，会自动启动所有master的namenode和slave1,slave2的datanode：

[email protected]:~/hadoop-2.4.1$ ./sbin/start-dfs.sh
Starting namenodes on [master] 
The authenticity of host ‘master (192.168.174.160)‘ can‘t be established. 
ECDSA key fingerprint is 1f:c0:2a:ed:c1:7b:6e:26:46:e3:c3:b6:87:bb:99:42. 
Are you sure you want to continue connecting (yes/no)? yes 
master: Warning: Permanently added ‘master,192.168.174.160‘ (ECDSA) to the list of known hosts. 
[email protected]‘s password: 
master: mkdir: 无法创建目录"/opt/hadoop-2.4.1/logs": 权限不够 
master: chown: 无法访问"/opt/hadoop-2.4.1/logs": 没有那个文件或目录 
master: starting namenode, logging to /opt/hadoop-2.4.1/logs/hadoop-hadoop-namenode-master.out 
master: /opt/hadoop-2.4.1/sbin/hadoop-daemon.sh: 行 151: /opt/hadoop-2.4.1/logs/hadoop-hadoop-namenode-master.out: 没有那个文件或目录 
master: head: 无法打开"/opt/hadoop-2.4.1/logs/hadoop-hadoop-namenode-master.out" 读取数据: 没有那个文件或目录 
master: /opt/hadoop-2.4.1/sbin/hadoop-daemon.sh: 行 166: /opt/hadoop-2.4.1/logs/hadoop-hadoop-namenode-master.out: 没有那个文件或目录 
master: /opt/hadoop-2.4.1/sbin/hadoop-daemon.sh: 行 167: /opt/hadoop-2.4.1/logs/hadoop-hadoop-namenode-master.out: 没有那个文件或目录 
The authenticity of host ‘slave2 (192.168.174.162)‘ can‘t be established. 
ECDSA key fingerprint is 1f:c0:2a:ed:c1:7b:6e:26:46:e3:c3:b6:87:bb:99:42. 
Are you sure you want to continue connecting (yes/no)? The authenticity of host ‘slave1 (192.168.174.161)‘ can‘t be established. 
ECDSA key fingerprint is 1f:c0:2a:ed:c1:7b:6e:26:46:e3:c3:b6:87:bb:99:42. 
Are you sure you want to continue connecting (yes/no)? yes 
slave2: Warning: Permanently added ‘slave2,192.168.174.162‘ (ECDSA) to the list of known hosts. 
[email protected]‘s password: Please type ‘yes‘ or ‘no‘: 
slave1: Warning: Permanently added ‘slave1,192.168.174.161‘ (ECDSA) to the list of known hosts. 
[email protected]‘s password: 
slave2: mkdir: 无法创建目录"/opt/hadoop-2.4.1/logs": 权限不够 
slave2: chown: 无法访问"/opt/hadoop-2.4.1/logs": 没有那个文件或目录 
slave2: starting datanode, logging to /opt/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave2.out 
slave2: /opt/hadoop-2.4.1/sbin/hadoop-daemon.sh: 行 151: /opt/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave2.out: 没有那个文件或目录 
slave2: head: 无法打开"/opt/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave2.out" 读取数据: 没有那个文件或目录 
slave2: /opt/hadoop-2.4.1/sbin/hadoop-daemon.sh: 行 166: /opt/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave2.out: 没有那个文件或目录 
slave2: /opt/hadoop-2.4.1/sbin/hadoop-daemon.sh: 行 167: /opt/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave2.out: 没有那个文件或目录

[email protected]:~/hadoop-2.4.1$ ./sbin/start-dfs.sh

【出现问题】

mkdir: 无法创建目录"/home/hadoop/hadoop-2.4.1/logs": 权限不够

【解决办法】

在master上都执行命令：

[email protected]:~$ sudo chown -R hadoop:hadoop hadoop-2.4.1/

slave1和slave2同样需要执行。

重新启动HDFS

[email protected]:~/hadoop-2.4.1$ ./sbin/start-dfs.sh 
Starting namenodes on [master] 
[email protected]‘s password: 
master: starting namenode, logging to /home/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-namenode-master.out 
hadoop[email protected]‘s password: [email protected]‘s password: 
slave1: starting datanode, logging to /home/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave1.out 

[email protected]‘s password: slave2: Permission denied, please try again. 

slave2: starting datanode, logging to /home/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-datanode-slave2.out 
Starting secondary namenodes [master] 
[email protected]‘s password: 
master: starting secondarynamenode, logging to /home/hadoop/hadoop-2.4.1/logs/hadoop-hadoop-secondarynamenode-master.out

[email protected]:~/hadoop-2.4.1$ ./sbin/start-dfs.sh

检查Hadoop集群是否安装好了，在master上面运行jps，如果有NameNode这个进程，说明master安装好了：

[email protected]:~/hadoop-2.4.1$ jps 
31711 SecondaryNameNode 
31464 NameNode 
31857 Jps

在slave1上面运行jps，如果有DataNode这个进程，说明slave1安装好了。

[email protected]:~$ jps 
5529 DataNode 
5610 Jps

在slave2上面运行jps，如果有DataNode这个进程，说明slave1安装好了。

[email protected]:~$ jps 
8119 Jps 
8035 DataNode

<-----------------------------安装完成----------------------------->

时间： 2024-08-09 06:33:41

VMware下Hadoop 2.4.1完全分布式集群平台安装与设置的相关文章

Hadoop及Zookeeper+HBase完全分布式集群部署

Hadoop及HBase集群部署 一. 集群环境 系统版本 虚拟机:内存 16G CPU 双核心 系统: CentOS-7 64位 系统下载地址: http://124.202.164.6/files/417500000AB646E7/mirrors.163.com/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1708.iso 软件版本 hadoop-2.8.1.tar.gz hbase-1.3.1-bin.tar.gz zookeeper-3.4.10.t

Hadoop伪分布式集群的安装部署

一.如何为虚拟机做快照? 1.什么是快照? 快照就是对当前的虚拟机状态进行拍照,保留虚拟机当前状态的操作信息. 2.为什么要为虚拟机做快照? 第一:为克隆不同状态的虚拟机提前做准备. 第二:当对虚拟机的某些操作执行错误而且改正比较麻烦的时候,可以切换到之前正常的虚拟机状态重新进行相关的操作. 3.如何为虚拟机做快照? (1)选择要克隆的虚拟机,然后“右键”,选择“快照”,然后选择“拍摄快照”. (2)可以为快照取个名称,也可以为虚拟机当前的状态做个描述,然后点击“拍摄快照”. 4.如何转到某一特

MapR分布式集群的安装配置及shell自动化脚本

MapR的分布式集群安装过程还是很艰难的,远远没有计划中的简单.本人总结安装配置,由于集群有很多机器,手动每台配置是很累的,编写了一个自动化配置脚本,下面以脚本为主线叙述(脚本并不完善,后续继续完善中). 编写脚本需要shell基础的,还要掌握awk和sed等统计手段为佳.操作系统是CentOS7. 准备:安装CentOS操作系统并分区 分区: / 50G /home 50G /boot 500M /var 50G /swap 8G /disk 305G(剩余全部,用于MapR的文件管理系统使用

VMware 虚拟机安装 hadoop 2.6.0 完全分布式集群

最近连着搭了两次hadoop的集群,搭建的时候也碰到了一些问题,因为之前对linux 不熟悉,经常遇到各种问题和命令忘记写,幸亏有度娘谷哥,这里做一个记录 下次使用的时候用的上 计算机的配置 计算机: G3440 3.3G 双核 8G内存 虚拟机: vmware workstation 12 pro 系统: centos6.5 节点: 192.168.133.33 master.hadoop 192.168.1.151 slave1.hadoop 192.168.1.151 slave2.had

mac 下搭建Elasticsearch 5.4.3分布式集群

一.集群角色 多机集群中的节点可以分为master nodes和data nodes,在配置文件中使用Zen发现(Zen discovery)机制来管理不同节点.Zen发现是ES自带的默认发现机制,使用多播发现其它节点.只要启动一个新的ES节点并设置和集群相同的名称这个节点就会被加入到集群中. Elasticsearch集群中有的节点一般有三种角色:master node.data node和client node. master node:master几点主要用于元数据(metadata)的处

Linux下部署Kafka分布式集群，安装与测试

注意:部署Kafka之前先部署环境JAVA.Zookeeper 准备三台CentOS_6.5_x64服务器,分别是:IP: 192.168.0.249 dbTest249 Kafka IP: 192.168.0.250 Other250 Kafka IP: 192.168.0.251 webTest251 Kafka 一.安装Kafka # tar zxvf kafka_2.9.1-0.8.2.2.tgz -C /usr/local/ # cd /usr/local/kafka_2.9.1-0.

使用Docker在本地搭建Hadoop分布式集群

学习Hadoop集群环境搭建是Hadoop入门必经之路.搭建分布式集群通常有两个办法: 要么找多台机器来部署(常常找不到机器) 或者在本地开多个虚拟机(开销很大,对宿主机器性能要求高,光是安装多个虚拟机系统就得搞半天……). 那么,问题来了! 有没有更有可行性的办法? 提到虚拟化,Docker最近很是火热!不妨拿来在本地做虚拟化,搭建Hadoop的伪分布式集群环境.虽然有点大材小用,但是学习学习,练练手也是极好的. 文章比较长,建议先倒杯水,听我慢慢到来…… 先说一下我的思路吧: 先使用Dock

spark1.6分布式集群环境搭建

1. 概述 本文是对spark1.6.0分布式集群的安装的一个详细说明,旨在帮助相关人员按照本说明能够快速搭建并使用spark集群. 2. 安装环境 本安装说明的示例环境部署如下: IP 外网IP hostname 备注 10.47.110.38 120.27.153.137 iZ237654q6qZ Master.Slaver 10.24.35.51 114.55.56.190 iZ23pd81xqaZ Slaver 10.45.53.136 114.55.11.55 iZ23mr5ukpzZ

高效搭建Spark完全分布式集群

写在前面一: 本文详细总结Spark分布式集群的安装步骤,帮助想要学习Spark的技术爱好者快速搭建Spark的学习研究环境. 写在前面二: 使用软件说明 约定,Spark相关软件存放目录:/usr/local/yujianxin/spark, Hadoop相关软件存放目录: /home/yujianxin/hadoop. Spark集群示意图 一.具体安装步骤 1.安装jdk 2.安装Hadoop集群,参考http://blog.csdn.net/aaronhadoop/article/det