2.1 Hadoop Eclipse Plugin 配置及安装

Hadoop Eclipse 开发工具 主要分为

1、根据Hadoop版本生成插件

2、安装Hadoop Eclipse插件

3、配置Hadoop目录

4、配置Hadoop连接

5、新一个MapReduce工程

WordCount.java

MapReduce——WordCount问题总结

参考：http://blog.sina.com.cn/s/blog_7fcb1aef0100zpux.html

正成功输入出后信息:

14/05/21 23:06:47 INFO
input.FileInputFormat: Total input paths to process : 2
14/05/21
23:06:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
14/05/21 23:06:47 WARN snappy.LoadSnappy: Snappy
native library not loaded
14/05/21 23:06:48 INFO mapred.JobClient: Running
job: job_201405220635_0009
14/05/21 23:06:49 INFO mapred.JobClient: map 0%
reduce 0%
14/05/21 23:06:59 INFO
mapred.JobClient: map 50% reduce 0%
14/05/21
23:07:00 INFO mapred.JobClient: map 100% reduce 0%
14/05/21
23:07:09 INFO mapred.JobClient: map 100% reduce 33%
14/05/21
23:07:11 INFO mapred.JobClient: map 100% reduce 100%
14/05/21
23:07:13 INFO mapred.JobClient: Job complete:
job_201405220635_0009
14/05/21
23:07:13 INFO mapred.JobClient: Counters: 29
14/05/21
23:07:13 INFO mapred.JobClient: Job Counters
14/05/21
23:07:13 INFO mapred.JobClient: Launched reduce tasks=1
14/05/21 23:07:13 INFO mapred.JobClient:
SLOTS_MILLIS_MAPS=17386
14/05/21
23:07:13 INFO mapred.JobClient: Total time spent by all reduces waiting
after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient:
Total time spent by all maps waiting after reserving slots
(ms)=0
14/05/21 23:07:13 INFO
mapred.JobClient: Launched map tasks=2
14/05/21
23:07:13 INFO mapred.JobClient: Data-local map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient:
SLOTS_MILLIS_REDUCES=12160
14/05/21 23:07:13 INFO mapred.JobClient: File
Output Format Counters
14/05/21
23:07:13 INFO mapred.JobClient: Bytes Written=15
14/05/21
23:07:13 INFO mapred.JobClient: FileSystemCounters
14/05/21
23:07:13 INFO mapred.JobClient: FILE_BYTES_READ=52
14/05/21
23:07:13 INFO mapred.JobClient: HDFS_BYTES_READ=252
14/05/21
23:07:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=177419
14/05/21 23:07:13 INFO mapred.JobClient:
HDFS_BYTES_WRITTEN=15
14/05/21
23:07:13 INFO mapred.JobClient: File Input Format Counters
14/05/21 23:07:13 INFO mapred.JobClient:
Bytes Read=22
14/05/21 23:07:13
INFO mapred.JobClient: Map-Reduce Framework
14/05/21
23:07:13 INFO mapred.JobClient: Map output materialized
bytes=58
14/05/21 23:07:13 INFO
mapred.JobClient: Map input records=2
14/05/21
23:07:13 INFO mapred.JobClient: Reduce shuffle bytes=58
14/05/21 23:07:13 INFO mapred.JobClient:
Spilled Records=8
14/05/21
23:07:13 INFO mapred.JobClient: Map output bytes=38
14/05/21
23:07:13 INFO mapred.JobClient: CPU time spent (ms)=5610
14/05/21 23:07:13 INFO mapred.JobClient:
Total committed heap usage (bytes)=336338944
14/05/21
23:07:13 INFO mapred.JobClient: Combine input records=4
14/05/21 23:07:13 INFO mapred.JobClient:
SPLIT_RAW_BYTES=230
14/05/21
23:07:13 INFO mapred.JobClient: Reduce input records=4
14/05/21 23:07:13 INFO mapred.JobClient:
Reduce input groups=2
14/05/21
23:07:13 INFO mapred.JobClient: Combine output records=4
14/05/21 23:07:13 INFO mapred.JobClient:
Physical memory (bytes) snapshot=428146688
14/05/21
23:07:13 INFO mapred.JobClient: Reduce output records=2
14/05/21 23:07:13 INFO mapred.JobClient:
Virtual memory (bytes) snapshot=2140233728
14/05/21
23:07:13 INFO mapred.JobClient: Map output records=4

错误解决方法：

14/05/21 23:14:11 INFO mapred.JobClient:
Cleaning up the staging area
hdfs://192.168.1.53:9000/app/hadoop/hadoop/tmp/mapred/staging/hadoop/.staging/job_201405220635_0010
14/05/21 23:14:11 ERROR
security.UserGroupInformation: PriviledgedActionException as:hadoop
cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
newout already exists
Exception
in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output
directory newout already exists
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:973)
at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at
org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at
org.apache.hadoop.examples.WordCount.main(WordCount.java:93)

cause:org.apache.hadoop.mapred.FileAlreadyExistsException:
Output directory newout already exists

说明目录已经存在了，需要删除后再执行，命令如下：

hadoop fs -rmr newout

2.1 Hadoop Eclipse Plugin 配置及安装,布布扣,bubuko.com

时间： 2024-10-06 06:24:37

2.1 Hadoop Eclipse Plugin 配置及安装的相关文章

使用hadoop eclipse plugin提交Job并添加多个第三方jar

来自:http://heipark.iteye.com/blog/1171923 通过 "conf.set("tmpjars", jars);" 可以设置第三方jar,之前一直只是添加一个jar,运行OK,今天打算添加多个jar的时候发现mapreduce在运行时找不到 class(ClassNotFoundException),跟踪代码发现jar文件的确上传到了HDFS中,所以甚是无解,后来上传jar到 hdfs,然后使用DistributedCache.addF

安装Hadoop系列 — eclipse plugin插件编译安装配置

[一].环境参数 eclipse-java-kepler-SR2-linux-gtk-x86_64.tar.gz //现在改为eclipse-jee-kepler-SR2-linux-gtk-x86_64.tar.gz Hadoop1.0.3 Java 1.8.0 Ubuntu 12.04  64bit [二].安装配置 1.复制生成的 hadoop-eclipse-plugin-1.0.3.jar 到 eclipse/plugins 路径下,重启eclipse即可. 2.在eclipse菜单依

Hadoop eclipse plugin

我的eclipse是在win7上,hadoop在win7里的虚拟机里的ubuntu上,为了方便起见,想在eclipse上安装hadoop的插件,主要参考 https://my.oschina.net/muou/blog/408543,上面写得蛮详细的,配置完后,出现在上面所写的问题一,按照他所说的创建目录后,eclipse里的 DFS Locations 里面依旧显示Connection Refused.找到了官方文档: https://wiki.apache.org/hadoop/Connec

ubuntu 14.04 hadoop eclipse 0配置基本环境

动人的hadoop第二天.构造hadoop该环境还花了两天时间,在这里写自己配置的过程,我希望能帮助! 我将文中用到的全部资源都分享到了  这里,点开就能下载,不须要一个个的找啦! 当中有<Hadoop 技术内幕>这本书.第一章讲述了这个配置过程,可是不具体~ ---------------安装jdk------------------------------- 1. 下载jdk1.6.0_45 2.解压到opt目录下,配置/etc/profile.在文件尾部加上 #set java envi

windows Hadoop环境搭建之三---Hadoop eclipse Plugin

准备环境 先下载htrace-core-3.0.4.jar文件 官网链接: http://mvnrepository.com/artifact/org.htrace/htrace-core/3.0.4 copy到Hadoop的share/hadoop/common/lib目录下 避免出现错误找不到文件的错误. 下载hadoop2x-eclipse-plugin 官网地址: https://github.com/winghc/hadoop2x-eclipse-plugin 解压后,上传到 Hado

安装Maven并在eclipse中配置

由于最近面试的需要,所以想熟悉熟悉Spring MVC的基本流程,却发现好像忘得差不多了(之间去搞了两个月的Linux C 开发).直接使用eclipse自带的maven构建,还是会出莫名奇妙的问题,所以这里记录一下安装Apache Maven的过程, 毕竟Maven还是构建Java项目的利器. 1. 下载,安装 去Maven官网,下载最新版本到本地,解压到相应目录即可. 2. 配置 MAVEN_HOME 第一步:在系统环境变量里配置,新加MAVEN_HOME,比如 D:\Lib\apache-

Eclipse中配置python开发环境详解

1.下载python 安装包.python-2.6.6.msi.并安装.默认python会安装在C:\Python26下,查看环境变量,如果没有在path路径中写入则手动添加.打开一个dos窗口,验证python是否安装成功: C:\Documents and Settings\Administrator>python Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on win32 Typ

在eclipse中配置maven

在eclipse中配置maven 2012-08-27 22:53:30|  分类: java |  标签:eclipse  安装maven  |举报|字号 订阅 maven下载地址: http://maven.apache.org/download.cgi 默认STS和myeclipse都自带了maven的支持,如果使用eclipse开发maven项目,需要先安装m2eclipse这个eclipse的插件. 一.安装插件 Help - Install New Software... 在Inst

ubuntu环境下eclipse的安装以及hadoop插件的配置

ubuntu环境下eclipse的安装以及hadoop插件的配置 一.eclipse的安装 在ubuntu桌面模式下,点击任务栏中的ubuntu软件中心,在搜索栏搜索eclipse 注意:安装过程需要输入用户密码. 二.eclipse的配置 待eclipse安装好以后,在命令行输入whereis eclipse 找到eclipse的安装路径 在文件目录下找到eclipse中的插件目录 然后在打开一个文件目录窗口找到hadoop/contrib/eclipse-plugin中的eclipse插件—