一、修改Hadoop配置文件
1.修改core-site.xml文件
添加如下属性，让MapReduce作业可以使用Tachyon文件系统作为输入和输出
<property> <name>fs.tachyon.impl</name> <value>tachyon.hadoop.TFS</value> </property>
2.配置hadoop-env.sh
在hadoop-env.sh文件开头添加Tachyon客户端jar包路径的环境变量。
export HADOOP_CLASSPATH=/usr/local/tachyon/client/target/tachyon-client-0.5.0-jar-with-dependencies.jar
3.将修改后的配置文件同步到其它节点
[[email protected] hadoop]# scp hadoop-env.shnode2:/usr/local/hadoop/etc/hadoop/ hadoop-env.sh 100%3499 3.4KB/s 00:00 [[email protected] hadoop]# scp hadoop-env.shnode3:/usr/local/hadoop/etc/hadoop/ hadoop-env.sh 100%3499 3.4KB/s 00:00 [[email protected] hadoop]# scp core-site.xmlnode2:/usr/local/hadoop/etc/hadoop/ core-site.xml 100% 1421 1.4KB/s 00:00 [[email protected] hadoop]# scp core-site.xmlnode3:/usr/local/hadoop/etc/hadoop/ core-site.xml 100% 1421 1.4KB/s 00:00 [[email protected] hadoop]#
4.启动所有的zookeeper节点
[[email protected] hadoop]# zkServer.sh start JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [[email protected] hadoop]# ssh node2 zkServer.sh start JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [[email protected] hadoop]# ssh node3 zkServer.sh start JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [[email protected] hadoop]#
5.启动hadoop集群
[[email protected] hadoop]# pwd /usr/local/hadoop [[email protected] hadoop]# sbin/start-all.sh
6.启动Tachyon集群
[[email protected] hadoop]# tachyon-start.sh all Mount
7.启动node2上的Tachyon Master
[[email protected] ~]# tachyon-start.sh master Starting master @ node2 [[email protected] ~]#
8.查看进程
[[email protected] conf]# jps 21954 QuorumPeerMain 22398 JournalNode 24120 TachyonWorker 22765 NodeManager 22572 DFSZKFailoverController 22663 ResourceManager 24009 TachyonMaster 24354 Jps 22216 DataNode 22115 NameNode [[email protected] conf]# ssh node2 jps 15524 NameNode 16538 TachyonWorker 15880 NodeManager 15802 DFSZKFailoverController 16650 Jps 15592 DataNode 15456 QuorumPeerMain 15683 JournalNode 16598 TachyonMaster [[email protected] conf]# ssh node3 jps 9294 DataNode 9231 QuorumPeerMain 9382 JournalNode 10050 Jps 10007 TachyonWorker 9476 NodeManager [[email protected] conf]#
二、测试Mapreduce作业
1.上传测试文件到Tachyon
[[email protected] conf]# tachyon tfs copyFromLocal /etc/passwd /passwd Copied /etc/passwd to /passwd [[email protected] conf]# tachyon tfs tail /passwd tp:/sbin/nologin nobody:x:99:99:Nobody:/:/sbin/nologin
2.执行Mapreduce作业
[[email protected] hadoop]# pwd /usr/local/hadoop [[email protected] hadoop]# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount -libjars /usr/local/tachyon/client/target/tachyon-client-0.5.0-jar-with-dependencies.jar tachyon://node1:19998/passwd tachyon://node1:19998/out
3.查看执行完成后的输出结果
[[email protected] hadoop]# tachyon tfs cat/out/part-r-00000 Daemon:/var/cache/rpcbind:/sbin/nologin 1 Daemon:/var/run/pulse:/sbin/nologin 1 IPv4LL 1 NFS 1 SSH:/var/empty/sshd:/sbin/nologin 1 Service 1 Stack:/var/lib/avahi-autoipd:/sbin/nologin 1 System 1 User:/var/ftp:/sbin/nologin 1 User:/var/lib/nfs:/sbin/nologin 2
时间： 2024-07-30 15:09:44