Hadoop常见异常及其解决方式

1、Shell$ExitCodeException

现象：执行hadoop job时出现例如以下异常：

14/07/09 14:42:50 INFO mapreduce.Job: Task Id : attempt_1404886826875_0007_m_000000_1, Status : FAILED

Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:

org.apache.hadoop.util.Shell$ExitCodeException:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)

at org.apache.hadoop.util.Shell.run(Shell.java:418)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 1

原因及解决的方法：原因未知。

重新启动可恢复正常

2、libhadoop.so.1.0.0 which might have disabled stack guard

现象：Hadoop 2.2.0 - warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard.

原因及解决方法：

在/etc/profile中加入：

#hadoop configuration

export PATH=$PATH:/home/jediael/hadoop-2.4.1/bin:/home/jediael/hadoop-2.4.1/sbin

export HADOOP_HOME=/home/jediael/hadoop-2.4.1

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

此警告出现的原因是最后2项未加入。

3、Retrying connect to server: master166/10.252.48.166:9000. Already tried 0 time(s)

在datanode上运行hdfs相关命令时，出现下面错误：

[[email protected] ~]$ hadoop fs -ls /

14/08/31 15:00:37 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:38 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:39 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:40 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:41 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:42 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:43 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:44 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:45 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

14/08/31 15:00:46 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

ls: Call to master166/10.252.48.166:9000 failed on connection exception: java.net.ConnectException: Connection refused

出现以上错误，通常都是因为datanode无法连接到namenode所致，下面是一种情况：

/etc/hosts中存在127.0.0.1 *****的配置，如

127.0.0.1 localhost

将这些配置去掉，然后又一次格式化namenode。并重新启动hadoop进程就可以解决。

或者是下面原因：

hadoop安装完毕后，必需要用haddop namenode format格式化后，才干使用，假设重新启动机器

在启动hadoop后，用hadoop fs -ls命令老是报 10/09/25 18:35:29 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).的错误，

用jps命令。也看不不到namenode的进程。 必须再用命令hadoop namenode format格式化后。才干再使用

原因是：hadoop默认配置是把一些tmp文件放在/tmp文件夹下，重新启动系统后，tmp文件夹下的东西被清除。所以报错

解决方法：在conf/core-site.xml 中添加下面内容

<property>

<name>hadoop.tmp.dir</name>

<value>/var/log/hadoop/tmp</value>

<description>A base for other temporary directories</description>

</property>

重新启动hadoop后，格式化namenode就可以

4、Permission denied: user=liaoliuqing, access=WRITE, inode="":jediael:supergroup:rwxr-xr-x

原由于用户权限不足，能能訪写HDFS中的文件。

解决方式：

关闭hadoop权限。在hdfs-site.xml文件里加入

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

5、Incompatible namespaceIDs

2015-02-02 15:10:57,526 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

2015-02-02 15:10:57,543 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.

2015-02-02 15:10:57,543 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).

2015-02-02 15:10:57,544 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started

2015-02-02 15:10:57,699 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.

2015-02-02 15:10:58,090 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /mnt/tmphadoop/dfs/data: namenode namespaceID = 2017454015; datanode namespaceID = 1238467850

at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)

at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)

at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:414)

at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:321)

at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1712)

at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1651)

at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1669)

at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1795)

at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1812)

问题原因：

每次namenode format会又一次创建一个namenodeId,而${hadoop.tmp.dir}/dfs/data下包括了上次format下的id,当又一次运行namenode format时清空了namenode下的数据,可是没有清空datanode下的数据,所以造成namenode节点上的namespaceID与 datanode节点上的namespaceID不一致，从而导致从现上述异常，启动失败。

解决的方法：

（1）停止hadoop

stop-all.sh

（2）在各个slave中删除dfs.data.dir中的内容。

若此属性未改动。则其默认值为

<property>

<name>${dfs.data.dir}</name>

<value>${hadoop.tmp.dir}/dfs/data</value>

<description>Determines where on the local filesystem an DFS data node

should store its blocks.  If this is a comma-delimited

list of directories, then data will be stored in all named

directories, typically on different devices.

Directories that do not exist are ignored.

</description>

</property>

（3）又一次格式化namenode

hadoop namenode -format

然后start-all.sh启动hadoop

以上解决的方法须要将原有数据删除，若数据不能删除。则使用下面方法之中的一个：

（1）改动${dfs.data.dir}/current/VERSION文件。将datanode中的id改成与namenode中的id一致。

（2）改动${dfs.data.dir}

时间： 2024-08-05 02:22:25

Hadoop常见异常及其解决方式的相关文章

Hadoop常见异常及其解决方案

1.Shell$ExitCodeException 现象:运行hadoop job时出现如下异常: 14/07/09 14:42:50 INFO mapreduce.Job: Task Id : attempt_1404886826875_0007_m_000000_1, Status : FAILED Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoo

常见错误及解决方式

Javaweb项目启动 ,用浏览器查看时,常见错误及解决方式. 1.404错误 1.1 404数字是什么? -------是服务器执行完客户端的请求以后,返回给客户端的一个执行结果的状态编码 1.2  产生的原因 ------Web服务器(容器)根据请求地址找不到对应资源. 如: 地址错误(拼写不正确,字母大小写错误) web.xml文件中的两个<servlet-name>不一致 工程没有部署 Web应用程序部署结构没有遵守Servlet规范 1.3 解决方法 按照http://ip:port

Android 常见异常及解决办法

?  前言 本文主要记录 Android 的常见异常及解决办法,以备以后遇到相同问题时可以快速解决. 1.   java.lang.NullPointerException: Attempt to invoke virtual method 'void android.widget.TextView.setText(java.lang.CharSequence)' on a null object reference 1)   异常描述:这应该是一个很常见的异常,类似于 C# 中的"未将对象引用

hadoop常见错误及解决办法整理

1:Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out Answer:程序里面需要打开多个文件,进行分析,系统一般默认数量是1024,(用ulimit -a可以看到)对于正常使用是够了,但是对于程序来讲,就太少了.修改办法:修改2个文件.       /etc/security/limits.confvi /etc/security/limits.conf加上:* soft nofile 102400* hard nofi

Package should contain a content type part [M1.13]异常的解决方式

此异常的源代码如下: 1 public static Workbook create(InputStream in) throws 2 IOException,InvalidFormatException { 3 if (!in.markSupported()) { 4 in = new PushbackInputStream(in, 8); 5 } 6 if (POIFSFileSystem.hasPOIFSHeader(in)) { 7 return new HSSFWorkbook(in)

Maven常见异常及解决方法（本篇停更至16-4-12）

本篇文章记录了老猫在学习整合Maven和SSH过程中遇到的问题,有的问题可以解决.有的问题还不能解决. 方法不一定适合全部的环境.但绝对是本人常遇到的常见异常.在这里做一个笔记和记录,也分享给大家,希望大家多多给出见解. 假设有不同的见解,请依照编号写出自己的见解吧,老猫愿闻其详! 此文老猫原创.转载请加本文连接:http://blog.csdn.net/nthack5730/article/details/46633287 很多其它有关老猫的文章:http://blog.csdn.net/nt

Maven常见异常及解决方法（转）

异常1: [ERROR] Failed to execute goal on project biz_zhuhai: Could not resolve dependencies for project biz_zhuhai:biz_zhuhai:jar:0.0.1-SNAPSHOT: Failed to collect dependencies for [com.maywide.ibh:lib345:pom:1.0 (compile)]: Failed to read artifact des

Maven常见异常及解决方法

异常1: [ERROR] Failed to execute goal on project biz_zhuhai: Could not resolve dependencies for project biz_zhuhai:biz_zhuhai:jar:0.0.1-SNAPSHOT: Failed to collect dependencies for [com.maywide.ibh:lib345:pom:1.0 (compile)]: Failed to read artifact des

在Ubuntu下使用Eclispe连接hadoop时拒绝链接解决方式总结

用eclipse连接集群查看文件信息提示9000port拒绝连接的错误 Cannot connect to the Map/Reduce location: hadoop1.0.3 Call to ubuntu/192.168.1.111:9000 failed on connection exception: java.net.ConnectException: 拒绝连接 1.常见解决的方法:配置都非常正常,就是连接不了. 后来又一次配置hadoop location,把Map/Reduce