1 安装
#brew install hadoop
安装的是2.6.0,目录为/usr/local/Cellar/hadoop,如果想安装其他版本,则下载tar包解压即可。地址:http://mirrors.cnnic.cn/apache/hadoop/common/
2 配置
将hadoop可执行路径bin和sbin都配置到环境变量中
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/home
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/sbin/:$HADOOP_HOME/bin/:$PATH
接下来配置伪分布式:
(1)
配置core-site.xml
? hadoop cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
(2)hdfs的节点备份设置为1
配置hdfs-site.xml
? hadoop cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
(3)配置mapred-site.xml
? hadoop cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:9000/</value>
</property>
</configuration>
(4)将hadoop-env.sh里jdk路径配置好即可。
然后执行:格式化namenode
? hadoop namenode -format
? hadoop start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/03/29 22:35:41 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 7082. Stop it first.
localhost: datanode running as process 7163. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 7266. Stop it first.
15/03/29 22:35:47 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 7377. Stop it first.
localhost: nodemanager running as process 7466. Stop it first.
? hadoop jps
7163 DataNode
9619 Jps
7466 NodeManager
7377 ResourceManager
7082 NameNode
7266 SecondaryNameNode
? hadoop
可以看到五个节点都启动了。
3 运行例子wordcount
先创建目录input 以及文件并上传到hdfs
? libexec ls input
file1.txt file2.txt
?libexec hadoop fs -put input/file*.txt /input
? libexec hadoop fs -ls /input
15/03/29 22:46:37 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:46:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 shenyb supergroup 35 2015-03-29 11:02 /input/file1.txt
-rw-r--r-- 1 shenyb supergroup 38 2015-03-29 11:02 /input/file2.txt
? libexec hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output
运行如果不出错,结果输出到output目录如下:
? libexec hadoop fs -ls /output
15/03/29 22:56:29 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 shenyb supergroup 0 2015-03-29 12:14 /output/_SUCCESS
-rw-r--r-- 1 shenyb supergroup 67 2015-03-29 12:14 /output/part-r-00000
? libexec hadoop fs -cat /output/part-r-00000
15/03/29 22:56:50 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ai 2
am 3
am2 1
are 2
i 2
man 1
man2 1
shenya 2
who 2
you 1
you2 1
? libexec
时间: 2024-10-09 23:25:28