sparksql 提交到yarn出现 java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hive/ql/plan/TableDesc

出现这个问题，是因为spark的excutor执行的时候，缺少hive的依赖包，添加依赖包的参数是：

--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*"

下面是是一个例子：

spark-submit --class com.simple.spark.Test \
    --master yarn-client \
    --num-executors 2 \
    --driver-memory 600m \
    --executor-memory 600m \
    --executor-cores 1 \
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*" \
--jars /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hbase-common.jar test.jar

时间： 2024-11-05 11:46:55

sparksql 提交到yarn出现 java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hive/ql/plan/TableDesc的相关文章

hive 使用where条件报错 java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.ppd.ExprWalkerInfo.getConvertedNode

hadoop 版本 2.6.0 hive版本 1.1.1 错误: java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.ppd.ExprWalkerInfo.getConvertedNod…… 解决办法 1,修改Hive的配置文件conf/hive-site.xmlhive.optimize.ppd==false然后重启hadoop环境 2,hive命令行:执行set hive.optimize.ppd=false;

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps Hadoop2.6.0编程问题与解决

从hadoop 1.2.1升级到 Hadoop2.6.0,调试写代码,还是遇到一些问题的.这里记录一下,后续如果自己再遇到类似问题,那也好找原因了. 在eclipse里编译运行 WordCount,出现以下错误. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps at java.lang.ClassLoader.defineClass1(Native M

Maven项目java.lang.NoClassDefFoundError: Lorg/apache/log4j/Logger报错

本文转载自:http://www.javaweb1024.com/info/894.jspx maven管理的项目,里面已经引入了log4j的包 maven引入如下: <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.16</version> <scope>compile</scope>

hive Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive

hive启动报错 java.lang.ClassNotFoundException: org.apache.hadoop.mapred.MRVersion

[[email protected] hive-1.1.0-cdh5.7.0]$ hivewhich: no hbase in (/home/hadoop/app/hive-1.1.0-cdh5.7.0/bin:/home/hadoop/app/hadoop-2.8.1/bin:/usr/java/jdk1.8.0_45/bin:/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hadoop/bin)E

Hive JDBC:java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate anonymous

今天使用JDBC来操作Hive时,首先启动了hive远程服务模式:hiveserver2 &(表示后台运行),然后到eclipse中运行程序时出现错误: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://192.168.182.11:10000/default: Failed to open new session: java.lang.RuntimeException: org.

解决hiveserver2报错：java.io.IOException: Job status not available - Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

用户使用的sql: select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_0a3a743f0fe3; 下面做不同的测试: 1.beeline -u jdbc:hive2://0.0.0.0:10000 -e "select count( distinct patient_id ) from argus.table_aa000612_641cd8ce_ceff_4ea0_9b27_

hive中删除表的错误Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException

1:请看操作 [[email protected] hive-0.12.0-bin]$ hive Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties hive> show tables; OK t1 tianq tianqi Time taken: 3.338 seconds

关于java.lang.NoClassDefFoundError: org/apache/commons/collections/FastHashMap的错误解决办法

在JavaEE开发中,在把配置文件中的数据或用户表单提交上来的数据,封装在相应JavaBean的对象的对应属性中时:在实际开发中,使用第三方法工具包BeanUtils(commons-beanutils-xxx.jar 依赖于commons-logging-xx.jar). 如博主在JavaWeb工程中,导入的jar包:commons-beanutils-1.9.3.jar和commons-logging-1.2.jar 这是在运行时报错了:eption in thread "main"