mapreduce执行时报java.net.ConnectException

*/

.hljs {
display: block;
overflow-x: auto;
padding: 0.5em;
background: #F0F0F0;
}

/* Base color: saturation 0; */

.hljs,
.hljs-subst {
color: #444;
}

.hljs-comment {
color: #888888;
}

.hljs-keyword,
.hljs-attribute,
.hljs-selector-tag,
.hljs-meta-keyword,
.hljs-doctag,
.hljs-name {
font-weight: bold;
}

/* User color: hue: 0 */

.hljs-type,
.hljs-string,
.hljs-number,
.hljs-selector-id,
.hljs-selector-class,
.hljs-quote,
.hljs-template-tag,
.hljs-deletion {
color: #880000;
}

.hljs-title,
.hljs-section {
color: #880000;
font-weight: bold;
}

.hljs-regexp,
.hljs-symbol,
.hljs-variable,
.hljs-template-variable,
.hljs-link,
.hljs-selector-attr,
.hljs-selector-pseudo {
color: #BC6060;
}

/* Language color: hue: 90; */

.hljs-literal {
color: #78A960;
}

.hljs-built_in,
.hljs-bullet,
.hljs-code,
.hljs-addition {
color: #397300;
}

/* Meta color: hue: 200 */

.hljs-meta {
color: #1f7199;
}

.hljs-meta-string {
color: #4d99bf;
}

/* Misc effects */

.hljs-emphasis {
font-style: italic;
}

.hljs-strong {
font-weight: bold;
}

#MathJax_About {position: fixed; left: 50%; width: auto; text-align: center; border: 3px outset; padding: 1em 2em; background-color: #DDDDDD; color: black; cursor: default; font-family: message-box; font-size: 120%; font-style: normal; text-indent: 0; text-transform: none; line-height: normal; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; z-index: 201; border-radius: 15px; -webkit-border-radius: 15px; -moz-border-radius: 15px; -khtml-border-radius: 15px; box-shadow: 0px 10px 20px #808080; -webkit-box-shadow: 0px 10px 20px #808080; -moz-box-shadow: 0px 10px 20px #808080; -khtml-box-shadow: 0px 10px 20px #808080; filter: progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color=‘gray‘, Positive=‘true‘)}
#MathJax_About.MathJax_MousePost {outline: none}
.MathJax_Menu {position: absolute; background-color: white; color: black; width: auto; padding: 2px; border: 1px solid #CCCCCC; margin: 0; cursor: default; font: menu; text-align: left; text-indent: 0; text-transform: none; line-height: normal; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; z-index: 201; box-shadow: 0px 10px 20px #808080; -webkit-box-shadow: 0px 10px 20px #808080; -moz-box-shadow: 0px 10px 20px #808080; -khtml-box-shadow: 0px 10px 20px #808080; filter: progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color=‘gray‘, Positive=‘true‘)}
.MathJax_MenuItem {padding: 2px 2em; background: transparent}
.MathJax_MenuArrow {position: absolute; right: .5em; padding-top: .25em; color: #666666; font-size: .75em}
.MathJax_MenuActive .MathJax_MenuArrow {color: white}
.MathJax_MenuArrow.RTL {left: .5em; right: auto}
.MathJax_MenuCheck {position: absolute; left: .7em}
.MathJax_MenuCheck.RTL {right: .7em; left: auto}
.MathJax_MenuRadioCheck {position: absolute; left: 1em}
.MathJax_MenuRadioCheck.RTL {right: 1em; left: auto}
.MathJax_MenuLabel {padding: 2px 2em 4px 1.33em; font-style: italic}
.MathJax_MenuRule {border-top: 1px solid #CCCCCC; margin: 4px 1px 0px}
.MathJax_MenuDisabled {color: GrayText}
.MathJax_MenuActive {background-color: Highlight; color: HighlightText}
.MathJax_MenuDisabled:focus, .MathJax_MenuLabel:focus {background-color: #E8E8E8}
.MathJax_ContextMenu:focus {outline: none}
.MathJax_ContextMenu .MathJax_MenuItem:focus {outline: none}
#MathJax_AboutClose {top: .2em; right: .2em}
.MathJax_Menu .MathJax_MenuClose {top: -10px; left: -10px}
.MathJax_MenuClose {position: absolute; cursor: pointer; display: inline-block; border: 2px solid #AAA; border-radius: 18px; -webkit-border-radius: 18px; -moz-border-radius: 18px; -khtml-border-radius: 18px; font-family: ‘Courier New‘,Courier; font-size: 24px; color: #F0F0F0}
.MathJax_MenuClose span {display: block; background-color: #AAA; border: 1.5px solid; border-radius: 18px; -webkit-border-radius: 18px; -moz-border-radius: 18px; -khtml-border-radius: 18px; line-height: 0; padding: 8px 0 6px}
.MathJax_MenuClose:hover {color: white!important; border: 2px solid #CCC!important}
.MathJax_MenuClose:hover span {background-color: #CCC!important}
.MathJax_MenuClose:hover:focus {outline: none}
-->

* {display: table-row!important}
.MJXp-surd {vertical-align: top}
.MJXp-surd > * {display: block!important}
.MJXp-script-box > * {display: table!important; height: 50%}
.MJXp-script-box > * > * {display: table-cell!important; vertical-align: top}
.MJXp-script-box > *:last-child > * {vertical-align: bottom}
.MJXp-script-box > * > * > * {display: block!important}
.MJXp-mphantom {visibility: hidden}
.MJXp-munderover, .MJXp-munder {display: inline-table!important}
.MJXp-over {display: inline-block!important; text-align: center}
.MJXp-over > * {display: block!important}
.MJXp-munderover > *, .MJXp-munder > * {display: table-row!important}
.MJXp-mtable {vertical-align: .25em; margin: 0 .125em}
.MJXp-mtable > * {display: inline-table!important; vertical-align: middle}
.MJXp-mtr {display: table-row!important}
.MJXp-mtd {display: table-cell!important; text-align: center; padding: .5em 0 0 .5em}
.MJXp-mtr > .MJXp-mtd:first-child {padding-left: 0}
.MJXp-mtr:first-child > .MJXp-mtd {padding-top: 0}
.MJXp-mlabeledtr {display: table-row!important}
.MJXp-mlabeledtr > .MJXp-mtd:first-child {padding-left: 0}
.MJXp-mlabeledtr:first-child > .MJXp-mtd {padding-top: 0}
.MJXp-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 1px 3px; font-style: normal; font-size: 90%}
.MJXp-scale0 {-webkit-transform: scaleX(.0); -moz-transform: scaleX(.0); -ms-transform: scaleX(.0); -o-transform: scaleX(.0); transform: scaleX(.0)}
.MJXp-scale1 {-webkit-transform: scaleX(.1); -moz-transform: scaleX(.1); -ms-transform: scaleX(.1); -o-transform: scaleX(.1); transform: scaleX(.1)}
.MJXp-scale2 {-webkit-transform: scaleX(.2); -moz-transform: scaleX(.2); -ms-transform: scaleX(.2); -o-transform: scaleX(.2); transform: scaleX(.2)}
.MJXp-scale3 {-webkit-transform: scaleX(.3); -moz-transform: scaleX(.3); -ms-transform: scaleX(.3); -o-transform: scaleX(.3); transform: scaleX(.3)}
.MJXp-scale4 {-webkit-transform: scaleX(.4); -moz-transform: scaleX(.4); -ms-transform: scaleX(.4); -o-transform: scaleX(.4); transform: scaleX(.4)}
.MJXp-scale5 {-webkit-transform: scaleX(.5); -moz-transform: scaleX(.5); -ms-transform: scaleX(.5); -o-transform: scaleX(.5); transform: scaleX(.5)}
.MJXp-scale6 {-webkit-transform: scaleX(.6); -moz-transform: scaleX(.6); -ms-transform: scaleX(.6); -o-transform: scaleX(.6); transform: scaleX(.6)}
.MJXp-scale7 {-webkit-transform: scaleX(.7); -moz-transform: scaleX(.7); -ms-transform: scaleX(.7); -o-transform: scaleX(.7); transform: scaleX(.7)}
.MJXp-scale8 {-webkit-transform: scaleX(.8); -moz-transform: scaleX(.8); -ms-transform: scaleX(.8); -o-transform: scaleX(.8); transform: scaleX(.8)}
.MJXp-scale9 {-webkit-transform: scaleX(.9); -moz-transform: scaleX(.9); -ms-transform: scaleX(.9); -o-transform: scaleX(.9); transform: scaleX(.9)}
.MathJax_PHTML .noError {vertical-align: ; font-size: 90%; text-align: left; color: black; padding: 1px 3px; border: 1px solid}
-->

mapreduce执行时报java.net.ConnectException

解决方案

在执行mapreduce程序时出现java.net.ConnectException: 拒绝连接异常,在错误信息中看到" localhost:35334"字眼,怀疑是没有配置主机名的原因, (1)尝试对每台虚拟机设置主机名,同时更新ssh密钥信息。
(2)更改hdfs-site.xml和yarn-site.xml相关内容,将原来的ip号改为主机名形式,并在./etc/hadoop/slaves文件下追加slave节点的主机名,将这三个文件分别传送到slave节点上。
(3)删除master下namenode和tmp路径下的所有信息,删除slave节点下nodename和tmp路径下的所有信息.对master节点重新进行format

./bin/hadoop namenode -format

配置 core-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
 <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
        <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
        <description>host Single hostname that can be set in place of setting all yarn.resourcemanager*address resources. Results in default ports for ResourceManager components.</description>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        <description></description>
    </property>
</configuration>

slaves

node1
node2

错误详细信息

[[email protected] bin]# ./hadoop jar /home/wordcount.jar /home/test.txt /home/test/rs.txt
19/08/27 23:33:17 INFO client.RMProxy: Connecting to ResourceManager at /192.168.120.128:8032
19/08/27 23:33:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/08/27 23:33:35 INFO input.FileInputFormat: Total input files to process : 1
19/08/27 23:33:35 INFO mapreduce.JobSubmitter: number of splits:1
19/08/27 23:33:35 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/08/27 23:33:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1566918823206_0003
19/08/27 23:33:36 INFO impl.YarnClientImpl: Submitted application application_1566918823206_0003
19/08/27 23:33:37 INFO mapreduce.Job: The url to track the job: http://bogon:8088/proxy/application_1566918823206_0003/
19/08/27 23:33:37 INFO mapreduce.Job: Running job: job_1566918823206_0003
19/08/27 23:43:02 INFO mapreduce.Job: Job job_1566918823206_0003 running in uber mode : false
19/08/27 23:43:02 INFO mapreduce.Job:  map 0% reduce 0%
19/08/27 23:43:03 INFO mapreduce.Job: Job job_1566918823206_0003 failed with state FAILED due to: Application application_1566918823206_0003 failed 2 times due to Error launching appattempt_1566918823206_0003_000002. Got exception: java.net.ConnectException: Call From bogon/192.168.120.128 to localhost:35334 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor44.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
    at org.apache.hadoop.ipc.Client.call(Client.java:1453)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    at com.sun.proxy.$Proxy82.startContainers(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128)
    at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at com.sun.proxy.$Proxy83.startContainers(Unknown Source)
    at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122)
    at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:311)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒绝连接
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
    at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:412)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1568)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    ... 19 more
. Failing the application.
19/08/27 23:43:03 INFO mapreduce.Job: Counters: 0

参照文档

解决Host key verification failed

原文地址：https://www.cnblogs.com/zhengzuozhanglina/p/11421647.html

时间： 2024-10-19 19:27:13

mapreduce执行时报java.net.ConnectException的相关文章

ERROR security.UserGroupInformation: PriviledgedActionException + java.net.ConnectException解决办法

HADOOP运行mr程序时报错: 15/05/18 19:25:33 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 15/05/18 19:25:34 INFO ipc.Client: Retrying connect to server: nwj5/192.168.11.1

Lein: Exception in thread &quot;Thread-3&quot; java.net.ConnectException: Connection refused

leiningen Leiningen是你的主要工具, 它用于: 启动一个 REPL 下载+安装类库 运行你的程序 启动一个服务器, 运行你所写的webapps 安装 brew install leiningen 打开repl lein repl 12345678910111213 lein replnREPL server started on port 50438 on host 127.0.0.1 - nrepl://127.0.0.1:50438REPL-y 0.3.5, nREPL 0

Hadoop MapReduce执行过程详解（带hadoop例子）

https://my.oschina.net/itblog/blog/275294 摘要: 本文通过一个例子,详细介绍Hadoop 的 MapReduce过程. 分析MapReduce执行过程 MapReduce运行的时候,会通过Mapper运行的任务读取HDFS中的数据文件,然后调用自己的方法,处理数据,最后输出.Reducer任务会接收Mapper任务输出的数据,作为自己的输入数据,调用自己的方法,最后输出到HDFS的文件中.整个流程如图: Mapper任务的执行过程详解 每个Mapper任

Mapreduce执行过程分析(基于Hadoop2.4)——(一)

1 概述 该瞅瞅MapReduce的内部运行原理了,以前只知道个皮毛,再不搞搞,不然怎么死的都不晓得.下文会以2.4版本中的WordCount这个经典例子作为分析的切入点,一步步来看里面到底是个什么情况. 2 为什么要使用MapReduce Map/Reduce,是一种模式,适合解决并行计算的问题,比如TopN.贝叶斯分类等.注意,是并行计算,而非迭代计算,像涉及到层次聚类的问题就不太适合了. 从名字可以看出,这种模式有两个步骤,Map和Reduce.Map即数据的映射,用于把一组键值对映射成另

Mapreduce执行过程分析(基于Hadoop2.4)——(二)

4.3 Map类 创建Map类和map函数,map函数是org.apache.hadoop.mapreduce.Mapper类中的定义的,当处理每一个键值对的时候,都要调用一次map方法,用户需要覆写此方法.此外还有setup方法和cleanup方法.map方法是当map任务开始运行的时候调用一次,cleanup方法是整个map任务结束的时候运行一次. 4.3.1 Map介绍 Mapper类是一个泛型类,带有4个参数(输入的键,输入的值,输出的键,输出的值).在这里输入的键为Object(默认是

tomcat停止失败 java.net.ConnectException: Connection refused

今天遇到个很郁闷的问题,使用catalina.sh stop 或者 shutdown.sh 关闭tomcat总是失败,总提示连接超时. 开始以为是自己tomcat配置的问题,重新部署上没更改过配置的tomcat,结果依然是失败. 后来使用telnet 127.0.0.1 到shutdown的端口 发现失败: tomcat停止的时候将SHUTDOWN指令发送给127.0.0.1:8005端口执行: 问题原来出在自己之前测试的时候把配置有127.0.0.1 IP的lo网卡给禁用了: 启用lo网卡,然

Hadoop学习之MapReduce执行过程详解

转自:http://my.oschina.net/itblog/blog/275294 分析MapReduce执行过程 MapReduce运行的时候,会通过Mapper运行的任务读取HDFS中的数据文件,然后调用自己的方法,处理数据,最后输出.Reducer任务会接收Mapper任务输出的数据,作为自己的输入数据,调用自己的方法,最后输出到HDFS的文件中.整个流程如图: Mapper任务的执行过程详解 每个Mapper任务是一个java进程,它会读取HDFS中的文件,解析成很多的键值对,经过我

Call From master/192.168.128.135 to master:8485 failed on connection exception: java.net.ConnectException: Connection refused

hadoop集群搭建了ha,初次启动正常,最近几天启动时偶尔发现,namenode1节点启动后一段时间(大约10几秒-半分钟左右),namenode1上namenode进程停掉,查看日志: 1 2017-08-28 21:54:37,617 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: slave1/192.168.128.136:8485. Already tried 9 time(s); retry policy

记录一次读取hdfs文件时出现的问题java.net.ConnectException: Connection refused

公司的hadoop集群是之前的同事搭建的,我(小白一个)在spark shell中读取hdfs上的文件时,执行以下指令 >>> word=sc.textFile("hdfs://localhost:9000/user/hadoop/test.txt") >>> word.first() 报错:java.net.ConnectException: Call From hadoop/133.0.123.130 to localhost:9000 fail