java执行spark查询hbase的jar包出现错误提示:ob aborted due to stage failure: Master removed our application: FAILED

执行java调用scala 打包后的jar时候出现异常

  

/14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:39 INFO AppClient$ClientActor: Executor updated: app-20150414235011-0003/9 is now EXITED (Command exited with code 1)
15/04/14 23:57:39 INFO SparkDeploySchedulerBackend: Executor app-20150414235011-0003/9 removed: Command exited with code 1
15/04/14 23:57:39 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED
15/04/14 23:57:39 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/04/14 23:57:39 INFO TaskSchedulerImpl: Cancelling stage 0
15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1234)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
    at akka.actor.ActorCell.invoke(ActorCell.scala:456)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

问题1:

/14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memor
分析:这个是内存不足?我spark-env.sh的配置文件信息如下
export JAVA_HOME=/home/hadoop/jdk1.7.0_75
export SCALA_HOME=/home/hadoop/scala-2.11.6
export HADOOP_HOME=/home/hadoop/hadoop-2.3.0-cdh5.0.2
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.3.0-cdh5.0.2/etc/hadoop
export SPARK_CLASSPATH=/home/hadoop/hbase-0.96.1.1-cdh5.0.2/lib/*
export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=17077
export SPARK_MASTER_WEBUI_PORT=18080

export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_INSTANCES=1

问题2:
15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
这句话的代码:
 val count = hbaseRDD.count()
    println("HBase RDD Count:" + count)
    hbaseRDD.cache()
问题3:
in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED有遇到过类似的或者知道怎么解决的可以留言下
时间: 2024-11-09 01:00:54

java执行spark查询hbase的jar包出现错误提示:ob aborted due to stage failure: Master removed our application: FAILED的相关文章

java应用程序远程登录linux并执行其命令(ssh jar包)

http://www.ganymed.ethz.ch/ssh2/在这个网址下载一个调用ssh和scp命令的jar包. 然后,就可以写程序了.将上面的jar包导入MyEclipse,下面是一个类的实例代码. package hh; import java.io.BufferedReader;     import java.io.IOException;     import java.io.InputStream;    import java.io.InputStreamReader; imp

java调用scala 查询hbase数据

问题:将scala打成jar包,提供给java调用,但是java一直提示找不到类 实现功能:利用spark查询hbase数据,然后提供给外部接口调用 我的方式:spark查询Hbase用scala实现(已经完成),将scala打成jar 包提供给java调用一直提示找不到scala 中的主类

java如何在项目依赖的jar包中查找冲突的类

原文:java如何在项目依赖的jar包中查找冲突的类 源代码下载地址:http://www.zuidaima.com/share/1785617759915008.htm package com.whaty.util; import java.util.*; import java.io.*; public class JarClassFind { public static int count = 0; static public void main(String[] args) { /* *

java导出excel不需要额外jar包

目前我知道的在java中导出Excel可以用poi或在jsp的文件头改变输出流.下面再介绍一种就用java基础包导出的Excel.导出的格式形如: 源码如下: package csvExcel; import java.io.FileWriter; import java.io.IOException; import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Date; import java

java项目导出为可运行jar包

1.选中要导出的项目, 右键->export,选中java下的JAR file->next,选择要导出的路径->next,finish. 2.选中导出的jar文件用压缩文件管理器打开 3.有个META-INF目录,把里面的MANIFEST.MF文件拖出来.此时MANIFEST文件内容应该只有下面这一行: <span style="font-size:24px;"> Manifest-Version:1.0</span> 4.修改MANIFEST

不要轻易在java ext 目录放任何三方jar包

今天在编写一个简单spi 应用demo的时候,在编译时总有一个其他的错误,如下: ERROR Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project userloginspi: Fatal error compiling: java.lang.NoSuchMethodError: com.google.common.collec

如果Android的jar包导入错误,怎么修改呢?

如果jar包导入错误,怎么修改呢? 右键工程---->properties---->Java Build Path --->Libraries-->选择android-support-v4.jar展开---->Editor--->External Folder

用Spark查询HBase中的表数据

java代码如下: package db.query; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.Res

Java之 将程序打包成jar包

准备材料: 1.java文件: Helloworld.java package com.ray; public class HelloWorld{ public static void main(String[] args) { System.out.println("Hello!jar"); } } 2.指定目录:建一个目录,来存放.class文件和.jar文件. 本文指定目录为 E:\myclass . 一.命令行形式 1.将java文件编译到指定目录: 语法格式为:javac -