centos安装hadoop(伪分布式)

在本机上装的CentOS 5.5 虚拟机，

软件准备：jdk 1.6 U26

hadoop:hadoop-0.20.203.tar.gz

ssh检查配置

Linux代码  

  1. [[email protected] ~]# ssh-keygen -t  rsa
  2. Generating public/private rsa key pair.
  3. Enter file in which to save the key (/root/.ssh/id_rsa):
  4. Created directory ‘/root/.ssh‘.
  5. Enter passphrase (empty for no passphrase):
  6. Enter same passphrase again:
  7. Your identification has been saved in /root/.ssh/id_rsa.
  8. Your public key has been saved in /root/.ssh/id_rsa.pub.
  9. The key fingerprint is:
  10. a8:7a:3e:f6:92:85:b8:c7:be:d9:0e:45:9c:d1:36:3b [email protected]
  11. [[email protected] ~]#
  12. [[email protected] ~]# cd ..
  13. [[email protected] /]# cd root
  14. [[email protected] ~]# ls
  15. anaconda-ks.cfg  Desktop  install.log  install.log.syslog
  16. [[email protected] ~]# cd .ssh
  17. [[email protected] .ssh]# cat id_rsa.pub > authorized_keys
  18. [[email protected] .ssh]#
  19. [[email protected] .ssh]# ssh localhost
  20. The authenticity of host ‘localhost (127.0.0.1)‘ can‘t be established.
  21. RSA key fingerprint is 41:c8:d4:e4:60:71:6f:6a:33:6a:25:27:62:9b:e3:90.
  22. Are you sure you want to continue connecting (yes/no)? yes
  23. Warning: Permanently added ‘localhost‘ (RSA) to the list of known hosts.
  24. Last login: Tue Jun 21 22:40:31 2011
  25. [[email protected] ~]#
[[email protected] ~]# ssh-keygen -t  rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory ‘/root/.ssh‘.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a8:7a:3e:f6:92:85:b8:c7:be:d9:0e:45:9c:d1:36:3b [email protected]
[[email protected] ~]#
[[email protected] ~]# cd ..
[[email protected] /]# cd root
[[email protected] ~]# ls
anaconda-ks.cfg  Desktop  install.log  install.log.syslog
[[email protected] ~]# cd .ssh
[[email protected] .ssh]# cat id_rsa.pub > authorized_keys
[[email protected] .ssh]# 

[[email protected] .ssh]# ssh localhost
The authenticity of host ‘localhost (127.0.0.1)‘ can‘t be established.
RSA key fingerprint is 41:c8:d4:e4:60:71:6f:6a:33:6a:25:27:62:9b:e3:90.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost‘ (RSA) to the list of known hosts.
Last login: Tue Jun 21 22:40:31 2011
[[email protected] ~]#

安装jdk

Linux代码  

  1. [[email protected] java]# chmod +x jdk-6u26-linux-i586.bin
  2. [[email protected] java]# ./jdk-6u26-linux-i586.bin
  3. ......
  4. ......
  5. ......
  6. For more information on what data Registration collects and
  7. how it is managed and used, see:
  8. http://java.sun.com/javase/registration/JDKRegistrationPrivacy.html
  9. Press Enter to continue.....
  10. Done.
[[email protected] java]# chmod +x jdk-6u26-linux-i586.bin
[[email protected] java]# ./jdk-6u26-linux-i586.bin
......
......
......
For more information on what data Registration collects and
how it is managed and used, see:
http://java.sun.com/javase/registration/JDKRegistrationPrivacy.html

Press Enter to continue.....

Done.

安装完成后生成文件夹：jdk1.6.0_26

  配置环境变量

Linux代码  

  1. [[email protected] java]# vi /etc/profile
  2. #添加如下信息
  3. # set java environment
  4. export JAVA_HOME=/usr/java/jdk1.6.0_26
  5. export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
  6. export PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
  7. export HADOOP_HOME=/usr/local/hadoop/hadoop-0.20.203
  8. export PATH=$PATH:$HADOOP_HOME/bin
  9. [[email protected] java]# chmod +x  /etc/profile
  10. [[email protected] java]# source  /etc/profile
  11. [[email protected] java]#
  12. [[email protected] java]# java -version
  13. java version "1.6.0_26"
  14. Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
  15. Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)
  16. [[email protected] java]#
[[email protected] java]# vi /etc/profile
#添加如下信息
# set java environment
export JAVA_HOME=/usr/java/jdk1.6.0_26
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
export HADOOP_HOME=/usr/local/hadoop/hadoop-0.20.203
export PATH=$PATH:$HADOOP_HOME/bin

[[email protected] java]# chmod +x  /etc/profile
[[email protected] java]# source  /etc/profile
[[email protected] java]#
[[email protected] java]# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)
[[email protected] java]#

修改hosts

Linuxa代码  

  1. [[email protected] conf]# vi /etc/hosts
  2. # Do not remove the following line, or various programs
  3. # that require network functionality will fail.
  4. 127.0.0.1               localhost.localdomain localhost
  5. ::1             localhost6.localdomain6 localhost6
  6. 127.0.0.1       namenode datanode01
[[email protected] conf]# vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
127.0.0.1       namenode datanode01

解压安装hadoop

Linux代码  

  1. [[email protected] hadoop]# tar zxvf hadoop-0.20.203.tar.gz
  2. ......
  3. ......
  4. ......
  5. hadoop-0.20.203.0/src/contrib/ec2/bin/image/create-hadoop-image-remote
  6. hadoop-0.20.203.0/src/contrib/ec2/bin/image/ec2-run-user-data
  7. hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-cluster
  8. hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-master
  9. hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-slaves
  10. hadoop-0.20.203.0/src/contrib/ec2/bin/list-hadoop-clusters
  11. hadoop-0.20.203.0/src/contrib/ec2/bin/terminate-hadoop-cluster
  12. [[email protected] hadoop]#
[[email protected] hadoop]# tar zxvf hadoop-0.20.203.tar.gz
......
......
......
hadoop-0.20.203.0/src/contrib/ec2/bin/image/create-hadoop-image-remote
hadoop-0.20.203.0/src/contrib/ec2/bin/image/ec2-run-user-data
hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-cluster
hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-master
hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-slaves
hadoop-0.20.203.0/src/contrib/ec2/bin/list-hadoop-clusters
hadoop-0.20.203.0/src/contrib/ec2/bin/terminate-hadoop-cluster
[[email protected] hadoop]#

  进入hadoop配置conf

Linux代码  

  1. ####################################
  2. [[email protected] conf]# vi hadoop-env.sh
  3. # 添加代码
  4. # set java environment
  5. export JAVA_HOME=/usr/java/jdk1.6.0_26
  6. #####################################
  7. [[email protected] conf]# vi core-site.xml
  8. <?xml version="1.0"?>
  9. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  10. <!-- Put site-specific property overrides in this file. -->
  11. <configuration>
  12. <property>
  13. <name>fs.default.name</name>
  14. <value>hdfs://namenode:9000/</value>
  15. </property>
  16. <property>
  17. <name>hadoop.tmp.dir</name>
  18. <value>/usr/local/hadoop/hadooptmp</value>
  19. </property>
  20. </configuration>
  21. #######################################
  22. [[email protected] conf]# vi hdfs-site.xml
  23. <?xml version="1.0"?>
  24. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  25. <!-- Put site-specific property overrides in this file. -->
  26. <configuration>
  27. <property>
  28. <name>dfs.name.dir</name>
  29. <value>/usr/local/hadoop/hdfs/name</value>
  30. </property>
  31. <property>
  32. <name>dfs.data.dir</name>
  33. <value>/usr/local/hadoop/hdfs/data</value>
  34. </property>
  35. <property>
  36. <name>dfs.replication</name>
  37. <value>1</value>
  38. </property>
  39. </configuration>
  40. #########################################
  41. [[email protected] conf]# vi mapred-site.xml
  42. <?xml version="1.0"?>
  43. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  44. <!-- Put site-specific property overrides in this file. -->
  45. <configuration>
  46. <property>
  47. <name>mapred.job.tracker</name>
  48. <value>namenode:9001</value>
  49. </property>
  50. <property>
  51. <name>mapred.local.dir</name>
  52. <value>/usr/local/hadoop/mapred/local</value>
  53. </property>
  54. <property>
  55. <name>mapred.system.dir</name>
  56. <value>/tmp/hadoop/mapred/system</value>
  57. </property>
  58. </configuration>
  59. #########################################
  60. [[email protected] conf]# vi masters
  61. #localhost
  62. namenode
  63. #########################################
  64. [[email protected] conf]# vi slaves
  65. #localhost
  66. datanode01
####################################
[[email protected] conf]# vi hadoop-env.sh
# 添加代码
# set java environment
  export JAVA_HOME=/usr/java/jdk1.6.0_26

#####################################
[[email protected] conf]# vi core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
   <property>
     <name>fs.default.name</name>
     <value>hdfs://namenode:9000/</value>
   </property>
   <property>
     <name>hadoop.tmp.dir</name>
     <value>/usr/local/hadoop/hadooptmp</value>
   </property>
</configuration>

#######################################
[[email protected] conf]# vi hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
     <name>dfs.name.dir</name>
     <value>/usr/local/hadoop/hdfs/name</value>
  </property>
  <property>
     <name>dfs.data.dir</name>
     <value>/usr/local/hadoop/hdfs/data</value>
  </property>
  <property>
     <name>dfs.replication</name>
     <value>1</value>
  </property>
</configuration>

#########################################
[[email protected] conf]# vi mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
     <name>mapred.job.tracker</name>
     <value>namenode:9001</value>
  </property>
  <property>
     <name>mapred.local.dir</name>
     <value>/usr/local/hadoop/mapred/local</value>
  </property>
  <property>
     <name>mapred.system.dir</name>
     <value>/tmp/hadoop/mapred/system</value>
  </property>
</configuration>

#########################################
[[email protected] conf]# vi masters
#localhost
namenode

#########################################
[[email protected] conf]# vi slaves
#localhost
datanode01

启动 hadoop

Linux代码  

  1. #####################<span style="font-size: small;">格式化namenode##############</span>
  2. [[email protected] bin]# hadoop namenode -format
  3. 11/06/23 00:43:54 INFO namenode.NameNode: STARTUP_MSG:
  4. /************************************************************
  5. STARTUP_MSG: Starting NameNode
  6. STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
  7. STARTUP_MSG:   args = [-format]
  8. STARTUP_MSG:   version = 0.20.203.0
  9. STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by ‘oom‘ on Wed May  4 07:57:50 PDT 2011
  10. ************************************************************/
  11. 11/06/23 00:43:55 INFO util.GSet: VM type       = 32-bit
  12. 11/06/23 00:43:55 INFO util.GSet: 2% max memory = 19.33375 MB
  13. 11/06/23 00:43:55 INFO util.GSet: capacity      = 2^22 = 4194304 entries
  14. 11/06/23 00:43:55 INFO util.GSet: recommended=4194304, actual=4194304
  15. 11/06/23 00:43:56 INFO namenode.FSNamesystem: fsOwner=root
  16. 11/06/23 00:43:56 INFO namenode.FSNamesystem: supergroup=supergroup
  17. 11/06/23 00:43:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
  18. 11/06/23 00:43:56 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
  19. 11/06/23 00:43:56 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
  20. 11/06/23 00:43:56 INFO namenode.NameNode: Caching file names occuring more than 10 times
  21. 11/06/23 00:43:57 INFO common.Storage: Image file of size 110 saved in 0 seconds.
  22. 11/06/23 00:43:57 INFO common.Storage: Storage directory /usr/local/hadoop/hdfs/name has been successfully formatted.
  23. 11/06/23 00:43:57 INFO namenode.NameNode: SHUTDOWN_MSG:
  24. /************************************************************
  25. SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
  26. ************************************************************/
  27. [[email protected] bin]#
  28. ###########################################
  29. [[email protected] bin]# ./start-all.sh
  30. starting namenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-localhost.localdomain.out
  31. datanode01: starting datanode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
  32. namenode: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out
  33. starting jobtracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-localhost.localdomain.out
  34. datanode01: starting tasktracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-localhost.localdomain.out
  35. [[email protected] bin]# jps
  36. 11971 TaskTracker
  37. 11807 SecondaryNameNode
  38. 11599 NameNode
  39. 12022 Jps
  40. 11710 DataNode
  41. 11877 JobTracker
#####################格式化namenode##############

[[email protected] bin]# hadoop namenode -format
11/06/23 00:43:54 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.203.0
STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by ‘oom‘ on Wed May  4 07:57:50 PDT 2011
************************************************************/
11/06/23 00:43:55 INFO util.GSet: VM type       = 32-bit
11/06/23 00:43:55 INFO util.GSet: 2% max memory = 19.33375 MB
11/06/23 00:43:55 INFO util.GSet: capacity      = 2^22 = 4194304 entries
11/06/23 00:43:55 INFO util.GSet: recommended=4194304, actual=4194304
11/06/23 00:43:56 INFO namenode.FSNamesystem: fsOwner=root
11/06/23 00:43:56 INFO namenode.FSNamesystem: supergroup=supergroup
11/06/23 00:43:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/06/23 00:43:56 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
11/06/23 00:43:56 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/06/23 00:43:56 INFO namenode.NameNode: Caching file names occuring more than 10 times
11/06/23 00:43:57 INFO common.Storage: Image file of size 110 saved in 0 seconds.
11/06/23 00:43:57 INFO common.Storage: Storage directory /usr/local/hadoop/hdfs/name has been successfully formatted.
11/06/23 00:43:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/
[[email protected] bin]# 

###########################################
[[email protected] bin]# ./start-all.sh
starting namenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-localhost.localdomain.out
datanode01: starting datanode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
namenode: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out
starting jobtracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-localhost.localdomain.out
datanode01: starting tasktracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-localhost.localdomain.out
[[email protected] bin]# jps
11971 TaskTracker
11807 SecondaryNameNode
11599 NameNode
12022 Jps
11710 DataNode
11877 JobTracker

  查看集群状态

Linux代码  

  1. [[email protected] bin]# hadoop dfsadmin  -report
  2. Configured Capacity: 4055396352 (3.78 GB)
  3. Present Capacity: 464142351 (442.64 MB)
  4. DFS Remaining: 464089088 (442.59 MB)
  5. DFS Used: 53263 (52.01 KB)
  6. DFS Used%: 0.01%
  7. Under replicated blocks: 0
  8. Blocks with corrupt replicas: 0
  9. Missing blocks: 0
  10. -------------------------------------------------
  11. Datanodes available: 1 (1 total, 0 dead)
  12. Name: 127.0.0.1:50010
  13. Decommission Status : Normal
  14. Configured Capacity: 4055396352 (3.78 GB)
  15. DFS Used: 53263 (52.01 KB)
  16. Non DFS Used: 3591254001 (3.34 GB)
  17. DFS Remaining: 464089088(442.59 MB)
  18. DFS Used%: 0%
  19. DFS Remaining%: 11.44%
  20. Last contact: Thu Jun 23 01:11:15 PDT 2011
  21. [[email protected] bin]#
[[email protected] bin]# hadoop dfsadmin  -report
Configured Capacity: 4055396352 (3.78 GB)
Present Capacity: 464142351 (442.64 MB)
DFS Remaining: 464089088 (442.59 MB)
DFS Used: 53263 (52.01 KB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 4055396352 (3.78 GB)
DFS Used: 53263 (52.01 KB)
Non DFS Used: 3591254001 (3.34 GB)
DFS Remaining: 464089088(442.59 MB)
DFS Used%: 0%
DFS Remaining%: 11.44%
Last contact: Thu Jun 23 01:11:15 PDT 2011

[[email protected] bin]#

  其他问题： 1

Linux代码  

  1. ####################启动报错##########
  2. [[email protected] bin]# ./start-all.sh
  3. starting namenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-localhost.localdomain.out
  4. The authenticity of host ‘datanode01 (127.0.0.1)‘ can‘t be established.
  5. RSA key fingerprint is 41:c8:d4:e4:60:71:6f:6a:33:6a:25:27:62:9b:e3:90.
  6. Are you sure you want to continue connecting (yes/no)? y
  7. Please type ‘yes‘ or ‘no‘: yes
  8. datanode01: Warning: Permanently added ‘datanode01‘ (RSA) to the list of known hosts.
  9. datanode01: starting datanode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
  10. <strong><span style="color: rgb(255, 0, 0);">datanode01: Unrecognized option: -jvm
  11. datanode01: Could not create the Java virtual machine.</span>
  12. </strong>
  13. namenode: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out
  14. starting jobtracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-localhost.localdomain.out
  15. datanode01: starting tasktracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-localhost.localdomain.out
  16. [[email protected] bin]# jps
  17. 10442 JobTracker
  18. 10533 TaskTracker
  19. 10386 SecondaryNameNode
  20. 10201 NameNode
  21. 10658 Jps
  22. ################################################
  23. [[email protected] bin]# vi hadoop
  24. elif [ "$COMMAND" = "datanode" ] ; then
  25. CLASS=‘org.apache.hadoop.hdfs.server.datanode.DataNode‘
  26. if [[ $EUID -eq 0 ]]; then
  27. HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
  28. else
  29. HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
  30. fi
  31. #http://javoft.net/2011/06/hadoop-unrecognized-option-jvm-could-not-create-the-java-virtual-machine/
  32. #改为
  33. elif [ "$COMMAND" = "datanode" ] ; then
  34. CLASS=‘org.apache.hadoop.hdfs.server.datanode.DataNode‘
  35. #  if [[ $EUID -eq 0 ]]; then
  36. #    HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
  37. #  else
  38. HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
  39. #  fi
  40. #或者换非root用户启动
  41. #启动成功
####################启动报错##########
[[email protected] bin]# ./start-all.sh
starting namenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-localhost.localdomain.out
The authenticity of host ‘datanode01 (127.0.0.1)‘ can‘t be established.
RSA key fingerprint is 41:c8:d4:e4:60:71:6f:6a:33:6a:25:27:62:9b:e3:90.
Are you sure you want to continue connecting (yes/no)? y
Please type ‘yes‘ or ‘no‘: yes
datanode01: Warning: Permanently added ‘datanode01‘ (RSA) to the list of known hosts.
datanode01: starting datanode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
datanode01: Unrecognized option: -jvm
datanode01: Could not create the Java virtual machine.



namenode: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out
starting jobtracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-localhost.localdomain.out
datanode01: starting tasktracker, logging to /usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-localhost.localdomain.out
[[email protected] bin]# jps
10442 JobTracker
10533 TaskTracker
10386 SecondaryNameNode
10201 NameNode
10658 Jps

################################################
[[email protected] bin]# vi hadoop
elif [ "$COMMAND" = "datanode" ] ; then
  CLASS=‘org.apache.hadoop.hdfs.server.datanode.DataNode‘
  if [[ $EUID -eq 0 ]]; then
    HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
  else
    HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
  fi

#http://javoft.net/2011/06/hadoop-unrecognized-option-jvm-could-not-create-the-java-virtual-machine/
#改为
elif [ "$COMMAND" = "datanode" ] ; then
  CLASS=‘org.apache.hadoop.hdfs.server.datanode.DataNode‘
#  if [[ $EUID -eq 0 ]]; then
#    HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
#  else
    HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
#  fi

#或者换非root用户启动
#启动成功

2，启动时要关闭防火墙

查看运行情况：

http://localhost:50070

Firefox显示代码  

  1. NameNode ‘localhost.localdomain:9000‘
  2. Started:    Thu Jun 23 01:07:18 PDT 2011
  3. Version:    0.20.203.0, r1099333
  4. Compiled:   Wed May 4 07:57:50 PDT 2011 by oom
  5. Upgrades:   There are no upgrades in progress.
  6. Browse the filesystem
  7. Namenode Logs
  8. Cluster Summary
  9. 6 files and directories, 1 blocks = 7 total. Heap Size is 31.38 MB / 966.69 MB (3%)
  10. Configured Capacity :   3.78 GB
  11. DFS Used    :   52.01 KB
  12. Non DFS Used    :   3.34 GB
  13. DFS Remaining   :   442.38 MB
  14. DFS Used%   :   0 %
  15. DFS Remaining%  :   11.44 %
  16. Live Nodes  :   1
  17. Dead Nodes  :   0
  18. Decommissioning Nodes   :   0
  19. Number of Under-Replicated Blocks   :   0
  20. NameNode Storage:
  21. Storage Directory   Type    State
  22. /usr/local/hadoop/hdfs/name IMAGE_AND_EDITS Active
NameNode ‘localhost.localdomain:9000‘
Started: 	Thu Jun 23 01:07:18 PDT 2011
Version: 	0.20.203.0, r1099333
Compiled: 	Wed May 4 07:57:50 PDT 2011 by oom
Upgrades: 	There are no upgrades in progress.

Browse the filesystem
Namenode Logs
Cluster Summary
6 files and directories, 1 blocks = 7 total. Heap Size is 31.38 MB / 966.69 MB (3%)
Configured Capacity	:	3.78 GB
DFS Used	:	52.01 KB
Non DFS Used	:	3.34 GB
DFS Remaining	:	442.38 MB
DFS Used%	:	0 %
DFS Remaining%	:	11.44 %
Live Nodes 	:	1
Dead Nodes 	:	0
Decommissioning Nodes 	:	0
Number of Under-Replicated Blocks	:	0

NameNode Storage:
Storage Directory	Type	State
/usr/local/hadoop/hdfs/name	IMAGE_AND_EDITS	Active

http://localhost:50030

Firefox显示代码  

  1. namenode Hadoop Map/Reduce Administration
  2. Quick Links
  3. * Scheduling Info
  4. * Running Jobs
  5. * Retired Jobs
  6. * Local Logs
  7. State: RUNNING
  8. Started: Thu Jun 23 01:07:30 PDT 2011
  9. Version: 0.20.203.0, r1099333
  10. Compiled: Wed May 4 07:57:50 PDT 2011 by oom
  11. Identifier: 201106230107
  12. Cluster Summary (Heap Size is 15.31 MB/966.69 MB)
  13. Running Map Tasks   Running Reduce Tasks    Total Submissions   Nodes   Occupied Map Slots  Occupied Reduce Slots   Reserved Map Slots  Reserved Reduce Slots   Map Task Capacity   Reduce Task Capacity    Avg. Tasks/Node Blacklisted Nodes   Graylisted Nodes    Excluded Nodes
  14. 0   0   0   1   0   0   0   0   2   2   4.00    0   0   0
  15. Scheduling Information
  16. Queue Name  State   Scheduling Information
  17. default     running     N/A
  18. Filter (Jobid, Priority, User, Name)
  19. Example: ‘user:smith 3200‘ will filter by ‘smith‘ only in the user field and ‘3200‘ in all fields
  20. Running Jobs
  21. none
  22. Retired Jobs
  23. none
  24. Local Logs
  25. Log directory, Job Tracker History This is Apache Hadoop release 0.20.203.0
namenode Hadoop Map/Reduce Administration
Quick Links

    * Scheduling Info
    * Running Jobs
    * Retired Jobs
    * Local Logs

State: RUNNING
Started: Thu Jun 23 01:07:30 PDT 2011
Version: 0.20.203.0, r1099333
Compiled: Wed May 4 07:57:50 PDT 2011 by oom
Identifier: 201106230107
Cluster Summary (Heap Size is 15.31 MB/966.69 MB)
Running Map Tasks	Running Reduce Tasks	Total Submissions	Nodes	Occupied Map Slots	Occupied Reduce Slots	Reserved Map Slots	Reserved Reduce Slots	Map Task Capacity	Reduce Task Capacity	Avg. Tasks/Node	Blacklisted Nodes	Graylisted Nodes	Excluded Nodes
0	0	0	1	0	0	0	0	2	2	4.00	0	0	0

Scheduling Information
Queue Name 	State 	Scheduling Information
default 	running 	N/A
Filter (Jobid, Priority, User, Name)
Example: ‘user:smith 3200‘ will filter by ‘smith‘ only in the user field and ‘3200‘ in all fields
Running Jobs
none
Retired Jobs
none
Local Logs
Log directory, Job Tracker History This is Apache Hadoop release 0.20.203.0

测试：

Linux代码  

  1. ##########建立目录名称##########
  2. [[email protected] bin]# hadoop fs -mkdir  testFolder
  3. ###############拷贝文件到文件夹中
  4. [[email protected] local]# ls
  5. bin  etc  games  hadoop  include  lib  libexec  sbin  share  src  SSH_key_file
  6. [[email protected] local]# hadoop fs -copyFromLocal SSH_key_file testFolder
  7. 进入web页面即可查看
##########建立目录名称##########
[[email protected] bin]# hadoop fs -mkdir  testFolder

###############拷贝文件到文件夹中
[[email protected] local]# ls
bin  etc  games  hadoop  include  lib  libexec  sbin  share  src  SSH_key_file
[[email protected] local]# hadoop fs -copyFromLocal SSH_key_file testFolder

进入web页面即可查看

参考：http://bxyzzy.blog.51cto.com/854497/352692

附：  准备FTP :yum install vsftpd (方便文件传输  和hadoop无关)

关闭防火墙：service iptables start

启动FTP：service vsftpd start

centos安装hadoop(伪分布式)

时间： 2024-09-28 22:45:42

centos安装hadoop(伪分布式)的相关文章

Ubuntu 安装hadoop 伪分布式

一.安装JDK  : http://www.cnblogs.com/E-star/p/4437788.html 二.配置SSH免密码登录1.安装所需软件        sudo apt-get install install ssh2.配置ssh免密码登录        ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa        cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys3.验证是否成功        ssh

安装Hadoop伪分布式踩过的坑

折腾了好几天的问题,终于解决了.虽然这次遇到的问题没在网络上找到答案,不过,解决问题的过程或多或少受到了网上一些答案的启发.下面我将分享自己遇到的问题,并提供解决方法. (一).相关资料 http://hadoop.apache.org/docs/r1.0.4/cn/quickstart.html #官网指导 http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html #shell命令 https://blog.csdn.net/jsutdoi

安装hadoop伪分布式

修改hosts cat /etc/hosts 127.0.0.1 mo.don.com 创建用户 useradd hadoop passwd hadoop sudo授权 visudo hadoop ALL=(ALL) NOPASSWD: ALL 切换用户 su - hadoop 下载解压 wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz tar xf hadoop-3.1.2.t

Window 10 WSL 下hadoop 伪分布式安装

Window 10 下的WSL子系统的安装可以参考之前的文章:window10 WSL子系统上设置redis开发环境 有时候,为了在window系统上进行hadoop开发和测试,可以在WSL上安装hadoop的伪分布式系统 因为子系统使用的是ubuntu 18.04,所以下面的安装以此为参考 下面是具体的配置步骤: 1:JDK的安装配置:这是必不可少的步骤 2:ssh的安装配置:Ubuntu SSH安装及配置免密码登录,安装完后可以用 ssh localhost来验证是否成功 3:hadoop伪

Hadoop伪分布式的搭建

实验平台: Virtual Box 4.3.24 CentOS7 JDK 1.8.0_60 Hadoop 2.6.0 Hadoop基本安装配置主要包括以下几个步骤: 1)创建Hadoop用户 2)安装Java 3)设置SSH登陆权限 4)单机安装配置 5)伪分布式安装配置 1.1 创建Hadoop用户 linux创建用户的命令是useradd,设置密码的命令是passwd 在CentOS下,首先我们通过useradd命令创建一个Hadoop用户组,它的密码也是Hadoop: useradd ha

Hadoop初体验：快速搭建Hadoop伪分布式环境

0.前言 本文旨在使用一个全新安装好的Linux系统从0开始进行Hadoop伪分布式环境的搭建,以达到快速搭建的目的,从而体验Hadoop的魅力所在,为后面的继续学习提供基础环境. 对使用的系统环境作如下说明: 操作系统:CentOS 6.5 64位 主机IP地址:10.0.0.131/24 主机名:leaf 用户名:root hadoop版本:2.6.5 jdk版本:1.7 可以看到,这里直接使用root用户,而不是按照大多数的教程创建一个hadoop用户来进行操作,就是为了达到快速搭建Had

Hadoop伪分布式搭建步骤

说明: 搭建环境是VMware10下用的是Linux CENTOS 32位,Hadoop:hadoop-2.4.1  JAVA :jdk7 32位:本文是本人在网络上收集的HADOOP系列视频所附带的课堂笔记(这里直接抄过来的,望原谅,我不是坏人),记录备忘. 开始前让我们简单修改下LINUX系统的JDK:一般情况下,我们都要将linux自带的OPENJDK卸载掉,然后安装SUN的JDK(下面的代码来源网络,请注意真实自己环境) <1> 查看已经安装的JAVA版本信息 # java -vers

阿里云服务器centos7.3下搭建hadoop伪分布式环境

一.软硬件环境 CentOS 7.2 64位 OpenJDK-1.8.0 Hadoop-2.7 二.安装SSH客户端 安装ssh: yum install openssh-clients openssh-server 安装完成后,使用以下命令测试: ssh localhost输入 root 账户的密码,如果可以正常登录,则说明SSH安装没有问题. 配置SSH免key登陆 hadoop是一个分布式系统,节点间通过ssh通信,为了避免在连接过程中人工输入密码,需要进行ssh免key登陆的配置,由于本

Hadoop伪分布式的部署

一.部署前的准备 (1)软件 一个空白的CentOS映像文件 Xshell(用于传输文件) (2)虚拟机配置 网络配置:网络适配器选择桥接 ip地址.子网掩码.网关.DNS服务器地址 =>可以在安装时的图形界面完成配置 =>也可以 vim /etc/sysconfig/network-script/ifcfg-ens33 修改 具体如下: IPADDR="192.168.124.x" (和网关在同一网段) NETMASK="255.255.255.0"