hadoop-1.2.1 伪分布配置

首先JDK安装及相关环境变量配置

# Java environment setting
JAVA_HOME=/usr/java/default
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
PATH=$JAVA_HOME/bin:$PATH
JRE_HOME=$JAVA_HOME/jre
export JAVA_HOME CLASSPATH PATH JRE_HOME

# Maven configuration
M2_HOME=/usr/local/apache-maven/default
PATH=$PATH:$M2_HOME/bin

# hadoop setting
HADOOP_HOME=/home/asn/opt/hadoop-1.2.1
PATH=$PATH:$HADOOP_HOME/bin

1. 修改主机名

[[email protected] opt]$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=vm-centos6-asn

在/etc/hosts文件中，添加主机名映射

[[email protected] ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 vm-centos6-asn
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

2. hadoop配置

1）core-site.xml

 1 <?xml version="1.0"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3
 4 <!-- Put site-specific property overrides in this file. -->
 5
 6 <configuration>
 7     <property>
 8         <name>fs.default.name</name>
 9         <value>hdfs://192.168.56.101:9000</value>
10     </property>
11
12     <property>
13         <name>hadoop.tmp.dir</name>
14         <value>/home/asn/opt/hadoop-tmp</value>
15     </property>
16 </configuration>

2) hdfs-site.xml

 1 <?xml version="1.0"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3
 4 <!-- Put site-specific property overrides in this file. -->
 5
 6 <configuration>
 7     <property>
 8         <name>dfs.data.dir</name>
 9         <value>/home/asn/opt/hadoop-data</value>
10     </property>
11
12     <property>
13         <name>dfs.replication</name>
14         <value>1</value>
15     </property>
16 </configuration>

3) mapred-site.xml

 1 <?xml version="1.0"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3
 4 <!-- Put site-specific property overrides in this file. -->
 5
 6 <configuration>
 7
 8     <property>
 9         <name>mapred.job.tracker</name>
10         <value>192.168.56.101:9001</value>
11     </property>
12 </configuration>

4)  设置hadoop-env.sh文件中的JAVA_HOME环境变量

[[email protected] conf]$ cat hadoop-env.sh
# Set Hadoop-specific environment variables here.
JAVA_HOME环境变量必须配置，其他环境变量可选。 当运行一个分布式配置时，最好配置该文件中的JAVA_HOME环境变量，以便于在远程节点上被正确定义。
# The only required environment variable is JAVA_HOME.  All others are optional.
# When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on remote nodes.

# The java implementation to use.  Required.
export JAVA_HOME=/usr/java/default

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server

3. 免密码登陆ssh配置

[[email protected] ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/asn/.ssh/id_rsa):   私钥存放文件
Created directory ‘/home/asn/.ssh‘.
Enter passphrase (empty for no passphrase):    直接回车
Enter same passphrase again:  直接回车
Your identification has been saved in /home/asn/.ssh/id_rsa.
Your public key has been saved in /home/asn/.ssh/id_rsa.pub.   公钥存放文件
The key fingerprint is:
6e:78:e0:30:10:b5:6b:fb:11:55:55:46:1e:b2:8f:57 [email protected]asn
The key‘s randomart image is:
+--[ RSA 2048]----+
|  ...     ..oo=  |
|   . .   .   = . |
|  . .   .   . . E|
|   . . .     o . |
|    = o S   . o  |
|   . = =     .   |
|    . + +        |
|     . +         |
|      .          |
+-----------------+
[[email protected]-centos6-asn ~]$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys   把公钥拷贝到认证文件中

[[email protected]-centos6-asn ~]$ ssh localhost
The authenticity of host ‘localhost (::1)‘ can‘t be established.
RSA key fingerprint is bd:c8:f0:20:87:be:ee:2f:05:6d:01:14:94:d7:c0:11.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost‘ (RSA) to the list of known hosts.  首次连接提示是否把localhost添加到已知的主机列表中
Last login: Tue Dec 23 14:14:43 2014 from localhost
[[email protected]-centos6-asn ~]$ exit
logout
Connection to localhost closed.

[[email protected]-centos6-asn ~]$ ssh localhost      直接免密码登录成功
Last login: Tue Dec 23 14:16:14 2014 from localhost
[[email protected]-centos6-asn ~]$ 

4. hadoop  namenode -format  格式化名字节点

5. start-all.sh 启动

[[email protected] ~]$ start-all.sh
starting namenode, logging to /home/asn/opt/hadoop-1.2.1/libexec/../logs/hadoop-asn-namenode-vm-centos6-asn.out
localhost: starting datanode, logging to /home/asn/opt/hadoop-1.2.1/libexec/../logs/hadoop-asn-datanode-vm-centos6-asn.out
localhost: starting secondarynamenode, logging to /home/asn/opt/hadoop-1.2.1/libexec/../logs/hadoop-asn-secondarynamenode-vm-centos6-asn.out
starting jobtracker, logging to /home/asn/opt/hadoop-1.2.1/libexec/../logs/hadoop-asn-jobtracker-vm-centos6-asn.out
localhost: starting tasktracker, logging to /home/asn/opt/hadoop-1.2.1/libexec/../logs/hadoop-asn-tasktracker-vm-centos6-asn.out
[[email protected]-centos6-asn ~]$ jps
14406 Jps
13823 NameNode
14101 SecondaryNameNode
14193 JobTracker
14319 TaskTracker
13942 DataNode