解决spark程序报错：Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]

报错信息：

09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.aggregate.TungstenAggregate.doExecute(TungstenAggregate.scala:80)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.Exchange.prepareShuffleDependency(Exchange.scala:164)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.Exchange$$anonfun$doExecute$1.apply(Exchange.scala:254)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.Exchange$$anonfun$doExecute$1.apply(Exchange.scala:248)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     ... 64 more
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO - Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at scala.concurrent.Await$.result(package.scala:107)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.joins.BroadcastHashJoin.doExecute(BroadcastHashJoin.scala:107)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.Project.doExecute(basicOperators.scala:46)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1.apply(TungstenAggregate.scala:86)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1.apply(TungstenAggregate.scala:80)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO -     ... 73 more

根据最后的Caused by信息和stack trace信息进行搜索，确定是broacast阶段超时，解决方法：

http://stackoverflow.com/questions/41123846/scala-spark-dataframes-join-java-util-concurrent-timeoutexception-futures-t

时间： 2024-10-10 12:39:09

解决spark程序报错：Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]的相关文章

[java][spark][spark streamming]java.util.concurrent.TimeoutException: Futures timed out

spark streamming 程序提交到yarn 上运行报错 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mnt/disk3/hadoop/yarn/local/filecache/491/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBin

异常-java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]

1 详细异常 java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227) at org.apac

多线程AsyncTask中使用Jsoup 报错Caused by: java.lang.NoClassDefFoundError: org.jsoup.Jsoup

代码如下,在多线程AsyncTask类中的doInBackground调用Jsoup protected String doInBackground(String... params) { // TODO Auto-generated method stub Log.i(TAG, "doInBackground(Params... params) called"); try { Document doc=Jsoup.connect(params[0]).get(); Elements

spring-framework-3.2.4.RELEASE 整合hibernate-release-4.3.5.Final报错Caused by: java.lang.NoClassDefFound

LZ今天整合spring-framework-3.2.4.RELEASE 整合hibernate-release-4.3.5.Final报错Caused by: java.lang.NoClassDefFound Caused by: java.lang.NoClassDefFoundError: org/hibernate/service/jta/platform/spi/JtaPlatform. 最后从国外一篇文章找到解决办法:原文地址如下:点击打开链接产生原因是: Hibernate4.

SQuirreL连接Phoenix报java.util.concurrent.TimeoutException

1.表象 java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.get(FutureTask.java:205) at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.awaitConnection(OpenConnectionCommand.java:113) at net.sourceforge.sq

IDEA Spark程序报错处理

错误一: 17/11/02 16:19:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V at Person.<init>(RDD_To_DataFrame.scala:8) at RDD_To_DataFrame$.$anonfun$main$2(RDD_To_DataFrame.sc

解决小程序报错 Page "pages/index/main" has not been registered yet.

在小程序开发中,会频繁遇到 Page "pages/index/main" has not been registered yet. 这种报错,意思就说指定的页面没有注册,找不到.遇到这种问题很可能不是代码问题,解决起来也真的是浪费时间,下面我介绍两种方式. 方法一:从源头处解决产生这种现象的原因就是我们在新添加某些页面的时候为了图省事.方便,直接到文件夹中复制.粘贴.改文件名.通过这种方式创建页面就有很大几率报错,所以从源头解决的话,就是按照下图所示在项目目录中新建文件夹.文

centOS7启动apache，报错Caused by: java.net.UnknownHostException: wshCentOS:

测试方法: ping 127.0.0.1 ping localhost 解决方案: vi /etc/hosts 把127.0.0.1和wshCentOS绑定就好按 i 进入修改模式,按 esc 退出修改模式: 输入 :wq 退出: 在用如下命令测试下: 原文地址:https://www.cnblogs.com/wsh1230/p/8127153.html

关于 java.util.concurrent 您不知道的 5 件事--转

第 1 部分 http://www.ibm.com/developerworks/cn/java/j-5things4.html Concurrent Collections 是 Java™ 5 的巨大附加产品,但是在关于注释和泛型的争执中很多 Java 开发人员忽视了它们.此外(或者更老实地说),许多开发人员避免使用这个数据包,因为他们认为它一定很复杂,就像它所要解决的问题一样. 事实上,java.util.concurrent 包含许多类,能够有效解决普通的并发问题,无需复杂工序.阅读本文,