初学hadoop,今日遇到的问题记录如下
1 util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
这只是一个警告,不影响使用,如果想消除这个警告,那就在正确的系统上重新编译hadoop源码,替换libhadoop.so.1.0.0,在lib/native文件下。
2
1 16/07/27 16:03:14 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.16.60:8032 2 3 16/07/27 16:03:14 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.16.60:8032 4 5 16/07/27 16:03:15 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 6 7 16/07/27 16:03:16 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 8 9 16/07/27 16:03:17 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 10 11 16/07/27 16:03:18 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 12 13 16/07/27 16:03:19 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 14 15 16/07/27 16:03:20 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 16 17 16/07/27 16:03:21 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 18 19 16/07/27 16:03:22 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 20 21 16/07/27 16:03:23 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 22 23 16/07/27 16:03:24 INFO ipc.Client: Retrying connect to server: master/192.168.16.60:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
连接不到master:8032,查看yarn是否开启,如果已经开启则更改yarn配置文件yarn-site.xml
<property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8031</value> </property>
此次过程中使用到Hadoop调试,下面记录下调试方法以便以后学习,来自:http://blog.csdn.net/mango_song/article/details/8502394
方法1:修改$HADOOP_CONF_DIR/log4j.properties文件 hadoop.root.logger=ALL,console
将该文件添加到项目classpath目录,如conf/log4j.properties
在代码中加入:PropertyConfigurator.configure("conf/log4j.properties");
方法2:开启:export HADOOP_ROOT_LOGGER=DEBUG,console
关闭:export HADOOP_ROOT_LOGGER=INFO,console
实时查看和修改Hadoop日志级别
Hadoop的日志界面可以通过Hadoop命令和Web界面来修改。
Hadoop命令格式:
hadoop daemonlog -getlevel <host:port> <name>
hadoop daemonlog --setlevel <host:port> <name> <level>
<name>为类名,如:TaskTracker
<level>为日志级别,如:debug和info等