准备工作
操作系统
CentOS 7
软件环境
- JDK 1.7.0_79 下载地址
- SSH，正常来说是系统自带的，若没有请自行搜索安装方法
关闭防火墙
systemctl stop firewalld.service #停止firewall systemctl disable firewalld.service #禁止firewall开机启动
设置HostName
[[email protected] ~]# hostname localhost
安装环境
安装JDK
[[email protected] ~]# tar -xzvf jdk-7u79-linux-x64.tar.gz
配置java环境变量
[[email protected] ~]# vi /etc/profile #添加如下配置 JAVA_HOME=/root/jdk1.7.0_79 PATH=$JAVA_HOME/bin:$PATH CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME export PATH export CLASSPATH
验证java
[[email protected] ~]# java -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
待输出以上内容时说明java已安装配置成功。
安装Hadoop
安装Hadoop 2.6.4
[[email protected] ~]# tar -xzvf hadoop-2.6.4.tar.gz
配置Hadoop环境变量
[[email protected] ~]# vim /etc/profile #添加以下配置 export HADOOP_HOME=/root/hadoop-2.6.4 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin [[email protected] ~]# vim /root/hadoop-2.6.4/etc/hadoop/hadoop-env.sh #修改以下配置 # The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes. # The java implementation to use. export JAVA_HOME=/root/jdk1.7.0_79
验证Hadoop
[[email protected] ~]# hadoop version Hadoop 2.6.4 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 5082c73637530b0b7e115f9625ed7fac69f937e6 Compiled by jenkins on 2016-02-12T09:45Z Compiled with protoc 2.5.0 From source with checksum 8dee2286ecdbbbc930a6c87b65cbc010 This command was run using /root/hadoop-2.6.4/share/hadoop/common/hadoop-common-2.6.4.jar
修改Hadoop配置文件
配置文件均存放在/root/hadoop-2.6.4/etc/hadoop
<!-- core-site.xml-->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
<!-- hdfs-site.xml -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
<!-- mapred-site.xml -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
<!-- yarn-site.xml -->
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
SSH免密码登陆
[[email protected] ~]# ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa [[email protected] ~]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
输入以下命令，如果不要求输入密码则表示配置成功：
[[email protected] ~]# ssh localhost Last login: Fri May 6 05:17:32 2016 from 192.168.154.1
执行Hadoop
格式化hdfs
[[email protected] ~]# hdfs namenode -format
启动NameNode，DataNode和YARN
[[email protected] ~]# start-dfs.sh Starting namenodes on [localhost] localhost: starting namenode, logging to /root/hadoop-2.6.4/logs/hadoop-root-namenode-localhost.out localhost: starting datanode, logging to /root/hadoop-2.6.4/logs/hadoop-root-datanode-localhost.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /root/hadoop-2.6.4/logs/hadoop-root-secondarynamenode-localhost.out [[email protected] ~]# start-yarn.sh starting yarn daemons starting resourcemanager, logging to /root/hadoop-2.6.4/logs/yarn-root-resourcemanager-localhost.out localhost: starting nodemanager, logging to /root/hadoop-2.6.4/logs/yarn-root-nodemanager-localhost.out
向hdfs上传测试文件
首先在/root/test中建立test1.txt和test2.txt，分别输入“hello world”和“hello hadoop”并保存。
使用如下命令将文件上传至hdfs的input目录中：
[[email protected] ~]# hadoop fs -put /root/test/ input [[email protected] ~]# hadoop fs -ls input Found 2 items -rw-r--r-- 1 root supergroup 12 2016-05-06 06:35 input/test1.txt -rw-r--r-- 1 root supergroup 13 2016-05-06 06:35 input/test2.txt
执行wordcount demo
输入以下命令并等待执行结果：
[[email protected] ~]# hadoop jar /root/hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount input output
16/05/06 06:44:15 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/05/06 06:44:16 INFO input.FileInputFormat: Total input paths to process : 2
16/05/06 06:44:17 INFO mapreduce.JobSubmitter: number of splits:2
16/05/06 06:44:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462530786445_0001
16/05/06 06:44:18 INFO impl.YarnClientImpl: Submitted application application_1462530786445_0001
16/05/06 06:44:18 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1462530786445_0001/
16/05/06 06:44:18 INFO mapreduce.Job: Running job: job_1462530786445_0001
16/05/06 06:44:33 INFO mapreduce.Job: Job job_1462530786445_0001 running in uber mode : false
16/05/06 06:44:33 INFO mapreduce.Job: map 0% reduce 0%
16/05/06 06:44:52 INFO mapreduce.Job: map 50% reduce 0%
16/05/06 06:44:53 INFO mapreduce.Job: map 100% reduce 0%
16/05/06 06:45:03 INFO mapreduce.Job: map 100% reduce 100%
16/05/06 06:45:03 INFO mapreduce.Job: Job job_1462530786445_0001 completed successfully
16/05/06 06:45:04 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=55
FILE: Number of bytes written=320242
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=249
HDFS: Number of bytes written=25
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=34487
Total time spent by all reduces in occupied slots (ms)=7744
Total time spent by all map tasks (ms)=34487
Total time spent by all reduce tasks (ms)=7744
Total vcore-milliseconds taken by all map tasks=34487
Total vcore-milliseconds taken by all reduce tasks=7744
Total megabyte-milliseconds taken by all map tasks=35314688
Total megabyte-milliseconds taken by all reduce tasks=7929856
Map-Reduce Framework
Map input records=2
Map output records=4
Map output bytes=41
Map output materialized bytes=61
Input split bytes=224
Combine input records=4
Combine output records=4
Reduce input groups=3
Reduce shuffle bytes=61
Reduce input records=4
Reduce output records=3
Spilled Records=8
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=364
CPU time spent (ms)=3990
Physical memory (bytes) snapshot=515538944
Virtual memory (bytes) snapshot=2588155904
Total committed heap usage (bytes)=296755200
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=25
File Output Format Counters
Bytes Written=25
查看执行结果
[[email protected] ~]# hadoop fs -ls output Found 2 items -rw-r--r-- 1 root supergroup 0 2016-05-06 06:45 output/_SUCCESS -rw-r--r-- 1 root supergroup 25 2016-05-06 06:45 output/part-r-00000 [[email protected] ~]# hadoop fs -cat output/part-r-00000 hadoop 1 hello 2 world 1
至此，Pseudo-Distributed就已经完成了。
时间： 2024-10-12 21:46:24