mac 下hadoop安装并运行例子

1 安装

#brew install hadoop

安装的是2.6.0,目录为/usr/local/Cellar/hadoop,如果想安装其他版本,则下载tar包解压即可。地址:http://mirrors.cnnic.cn/apache/hadoop/common/

2 配置

将hadoop可执行路径bin和sbin都配置到环境变量中

export HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/home
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/sbin/:$HADOOP_HOME/bin/:$PATH

接下来配置伪分布式:

(1)

配置core-site.xml

?  hadoop  cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

(2)hdfs的节点备份设置为1

配置hdfs-site.xml

?  hadoop  cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

(3)配置mapred-site.xml

?  hadoop  cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>hdfs://localhost:9000/</value>
  </property>
</configuration>

(4)将hadoop-env.sh里jdk路径配置好即可。

然后执行:格式化namenode

?  hadoop namenode -format
?  hadoop  start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/03/29 22:35:41 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 7082. Stop it first.
localhost: datanode running as process 7163. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 7266. Stop it first.
15/03/29 22:35:47 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 7377. Stop it first.
localhost: nodemanager running as process 7466. Stop it first.
?  hadoop  jps
7163 DataNode
9619 Jps
7466 NodeManager
7377 ResourceManager
7082 NameNode
7266 SecondaryNameNode
?  hadoop

可以看到五个节点都启动了。

3 运行例子wordcount

先创建目录input 以及文件并上传到hdfs

?  libexec  ls input
file1.txt file2.txt
 ?libexec hadoop fs -put input/file*.txt  /input
?  libexec  hadoop fs -ls /input
15/03/29 22:46:37 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:46:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 shenyb supergroup         35 2015-03-29 11:02 /input/file1.txt
-rw-r--r--   1 shenyb supergroup         38 2015-03-29 11:02 /input/file2.txt
?  libexec  hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output
运行如果不出错,结果输出到output目录如下:
?  libexec  hadoop fs -ls /output
15/03/29 22:56:29 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 shenyb supergroup          0 2015-03-29 12:14 /output/_SUCCESS
-rw-r--r--   1 shenyb supergroup         67 2015-03-29 12:14 /output/part-r-00000
?  libexec  hadoop fs -cat /output/part-r-00000
15/03/29 22:56:50 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ai  2
am  3
am2 1
are 2
i   2
man 1
man2    1
shenya  2
who 2
you 1
you2    1
?  libexec
时间: 2024-10-09 23:25:28

mac 下hadoop安装并运行例子的相关文章

Mac下删除安装的pkg

Mac下的安装和删除都比windows更加简单清晰,这点在dmg方式下非常明显,但很多时候我们会使用pkg来进行安装,这样的安装想删除就有点麻烦了. 比如,我安装了Golang这个pkg用于go语言的编译,安装后确实可以使用,但安装的是一个工具链,并不是一个应用软件.所以在Application目录下是没有内容的,你可以查到安装在/usr/local/go这个目录下,但是否所有的内容都安装在这个目录下呢?我删除这个pkg的目的是为了用源码直接编译出工具链来应用,如果还有其他一些配置文件遗漏,可能

Mac 下 docker安装

http://www.th7.cn/system/mac/201405/56653.shtml Mac 下 docker安装 以及 处理错误Cannot connect to the Docker daemon. Is 'docker -d' running on this host? 和 dial unix /var/run/docker.sock: no such file or directory 云主机可以选择系统镜像快速创建主机,这比虚拟机更便捷了,我们本地也可以这么做了,因为有了Do

Mac下webpack安装

最近开始接触构建工具webpack,公司电脑是 windows,而我自己的呢是mac.本来以为在自己电脑安装很简单,但是出了点问题,所以写出来分享下. 这里用npm的方式安装,首先你要安装node.js,这里献上地址   https://nodejs.org/en/download/ node.js 的安装我就不多说了,很简单和普通软件安装没什么区别. node.js 安装好后,就等于你安装好了node 和npm ,下面我们打开终端执行命令查看下: node -v[查看node版本].npm -

Mac下新安装的MySQL无法登陆root用户解决方法

一 设置MySQL命令行搜索路径 0.苹果->系统偏好设置->最下边点mysql 在弹出页面中 启动mysql服务 1.打开终端,输入: sudo vi ~/.bash_profile 如果已存在删除:  sudo rm -rf .bash_* 2.输入 i 3.然后粘贴以下内容 # mysql alias mysql='/usr/local/mysql/bin/mysql' alias mysqladmin='/usr/local/mysql/bin/mysqladmin' # ls ali

Mac下go安装

1.0 访问下载地址,32位系统下载go1.0.3.darwin-386.pkg,64位系统下载go1.0.3.darwin-amd64.pkg,双击下载文件,一路默认安装点击下一步,这个时候go已经安装到你的系统中,默认已经在PATH中增加了相应的~/go/bin, 2.0 这个时候打开命令行终端,输入go 3.0 如果出现go的Usage信息,那么说明go已经安装成功了:如果出现该命令不存在,那么可以检查一下自己的PATH环境变中是否包含了go的安装目录 Mac下go安装

flume 1.7在windows下的安装与运行

flume 1.7在windows下的安装与运行 一.安装 安装java,配置环境变量. 安装flume,flume的官网http://flume.apache.org/,下载地址,下载后直接解压即可. 二.运行 创建配置文件:在解压后的文件apache-flume-1.6.0-bin/conf下创建一个example.conf,内容如下. # example.conf: A single-node Flume configuration # Name the components on thi

Linux下RabbitMQ安装、运行与管理

Linux下RabbitMQ安装.运行与管理 安装erlang 安装参考官网 RabbitMQ的安装需要Erlang的基础环境,必须按照RabbitMQ Erlang版本要求进行安装. 关于Erlang官方的安装方式有三种: 官方制作的依赖软件包 Erlang Solutions的软件包(这个可以自定义yum库安装,本人自己下载安装) EPEL(“Enterprise Linux的额外软件包”) cd /usr/local/src/ mkdir rabbitmq cd rabbitmq //下载

Mac下 cobra安装

Mac下 cobra安装 1.配置.bash_profile export GOPATH=$PWD/go export GOBIN=$GOPATH/bin export PATH=$PATH:$GOBIN 2.在$GOPATH/src/ go get -v github.com/spf13/cobra/cobra 没有xxx 上网可能会出错 3.在$GOPATH/src/新建. golang.org/x/文件夹 git clone https://github.com/golang/sys.gi

Ubuntu-64位下hadoop安装

本文是hadoop学习系列的第一篇文章本文将介绍Ubuntu环境下JDK的安装以及hadoop的安装阅读本文需要有一定Linux命令基础 前期准备 安装环境:ubuntu-16.04.3-desktop 64bit所需软件:jdk-8u161-linux-x64 ?? hadoop-3.0.0 JDK与hadoop的下载 下载上面的JDK安装包至~/Downloads目录,切换至该目录并解压缩.下载上面的hadoop安装包至~/Downloads目录,切换至该目录并解压缩. 也可以通过以下方式下