idea连接spark集群报错解析:Caused by: java.lang.ClassCastException

cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.sql.execution.aggregate.SortAggregateExec.aggregateExpressions of type scala.collection.Seq in instance of org.apache.spark.sql.execution.aggregate.SortAggregateExec
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 4.0 failed 4 times, most recent failure: Lost task 8.3 in stage 4.0 (TID 332, 172.16.43.200, executor 1): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.sql.execution.aggregate.SortAggregateExec.aggregateExpressions of type scala.collection.Seq in instance of org.apache.spark.sql.execution.aggregate.SortAggregateExec

  当只指定了spark集群的地址,没有设定setJars这个参数,那么就会报以上这种错误,解决方法就是设置setjars这个参数,如下:

System.setProperty("hadoop.home.dir", "E:\\winutils")
    val conf = new SparkConf().setAppName("DetailRatio")
      .setMaster("spark://172.xx.xx.xx:7077")
   // .setMaster("local")
     .setJars(List("E:\\vense_work\\venseData\\out\\artifacts\\DetailRatio_jar\\venseData.jar"))
    //.set("spark.submit.deployMode", "client")

  

原文地址:https://www.cnblogs.com/zhnagqi-dream/p/11813578.html

时间: 2024-10-06 03:27:51

idea连接spark集群报错解析:Caused by: java.lang.ClassCastException的相关文章

eclipse连接远程Hadoop报错,Caused by: java.io.IOException: 远程主机强迫关闭了一个现有的连接。

eclipse连接远程Hadoop报错,Caused by: java.io.IOException: 远程主机强迫关闭了一个现有的连接.全部报错信息如下: Exception in thread "main" java.io.IOException: Call to hadoopmaster/192.168.1.180:9000 failed on local exception: java.io.IOException: 远程主机强迫关闭了一个现有的连接. at org.apach

mysql 保存emoji时报,数据库报错:Caused by: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x98\x82\xF0\x9F...' for column 'review' at row 1

错误原因:我们可以看到错误提示中的字符0xF0 0x9F 0x98 0x84 ,这对应UTF-8编码格式中的4字节编码(UTF-8编码规范).正常的汉字一般不会超过3个字节,为什么为出现4个字节呢?实际上是它对应的是智能手机输入法中的表情.那为什么会报错呢?因为mysql中的utf-8并不是真正意义上的utf-8,它只能存储1~3个字节长度的utf-8编码,如果想存储4个字节的必须用utf8mb4类型.不而要使用utf8mb4类型,首先要保证Mysql版本要不低于 MySQL 5.5.3. 常用

spark提示Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;

spark提示Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map; 起因 编写了一个处理两列是否相等的UDF,这两列的数据结构是一样的,但是结构比较复杂,如下: |-- list: array (nullable = true) | |-- element: map (conta

关于android使用ksoap2报Caused by: java.lang.ClassCastException: org.ksoap2.SoapFault cannot be cast to org.ksoap2.serialization.SoapObject

Caused by: java.lang.ClassCastException: org.ksoap2.SoapFault cannot be cast to org.ksoap2.serialization.SoapObject 报这种类似的错误的,困扰了我挺久.偶尔报不是一直报. 原来是少了一个判断,因为服务器每次返回的不一定是我们想要的结果. 见图 困惑了好久,最后在 stackoverflow 里面才找到答案. 大概就是服务器异常的时候会导致这个,难怪我程序会崩溃有些时候,比如某个地方本

CEPH -S集群报错TOO MANY PGS PER OSD

背景 集群状态报错,如下: # ceph -s cluster 1d64ac80-21be-430e-98a8-b4d8aeb18560 health HEALTH_WARN <-- 报错的地方 too many PGs per OSD (912 > max 300) monmap e1: 1 mons at {node1=109.105.115.67:6789/0} election epoch 4, quorum 0 node1 osdmap e49: 2 osds: 2 up, 2 in

maven项目中使用redis集群报错: java.lang.NumberFormatException: For input string: &quot;[email&#160;protected]&quot;

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [redis.clients.jedis.JedisCluster]: Constructor threw exception; nested exception is java.lang.NumberFormatException: For input string: "[email protected]" at

redis集群报错:(error) CLUSTERDOWN The cluster is down

更换了电脑,把原来的电脑上的虚拟机复制到了新电脑上,启动虚拟机上的centos系统,然后启动redis集群(redis5版本),发现集群可以启动,redis进程也有,但是连接集群中的任意节点就报错,如下 查看单个节点的集群配置信息: 发现是因为原来的集群配置信息导致的错误,需要将每个redis实例下的dump.rdb 和nodes.conf文件删除,然后重新创建集群 重新创建集群: 首先启动6个redis实例 2 .进入任意一个redis实例,执行集群创建命令: ./redis-cli --cl

ceph集群报错:HEALTH_ERR 1 pgs inconsistent; 1 scrub errors

报错信息如下: [[email protected] ~]# ceph health detail HEALTH_ERR 1 pgs inconsistent; 1 scrub errors; pg 2.37c is active+clean+inconsistent, acting [75,6,35] 1 scrub errors 报错信息总结: 问题PG:2.37c OSD编号:75,6,35 执行常规修复: ceph pg repair 2.37c 查看修复结果: [[email prot

Java客户端连接kafka集群报错

往kafka集群发送消息时,报错如下: page_visits-1: 30005 ms has passed since batch creation plus linger time 加入log4j.properties,设置为DEBUG级别,错误如下: 2017-06-03 17:33:31,417 DEBUG [org.apache.kafka.clients.NetworkClient] - Error connecting to node 2 at kafka-cluster-64bi