【原创】大叔经验分享（100）Atlas导入hive元数据

首先要有HIVE_HOME环境变量，

如果是apache，直接配置为解压目录；如果是CDH，设置如下：

# export HIVE_HOME=/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hive

执行导入

# bin/import-hive.sh
...
Failed to import Hive Meta Data!!!

报错，查看日志

# more logs/import-hive.log
2020-01-11 14:42:38,951 INFO  - [main:] ~ Looking for atlas-application.properties in classpath (ApplicationProperties:110)
2020-01-11 14:42:38,955 INFO  - [main:] ~ Looking for /atlas-application.properties in classpath (ApplicationProperties:115)
2020-01-11 14:42:38,956 INFO  - [main:] ~ Loading atlas-application.properties from null (ApplicationProperties:123)
2020-01-11 14:42:38,984 ERROR - [main:] ~ Import failed (HiveMetaStoreBridge:176)
org.apache.atlas.AtlasException: Failed to load application properties
    at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:134)
    at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:86)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:120)
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null
    at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259)
    at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
    at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:197)
    at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:284)
    at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:69)
    at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:125)
    ... 2 more

提示找不到atlas-application.properties，将其拷贝到hive conf目录

# cp conf/atlas-application.properties /etc/hive/conf/

再次执行

# bin/import-hive.sh
...
Enter username for atlas :- admin
Enter password for atlas :-
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/json/JacksonJaxbJsonProvider
    at org.apache.atlas.AtlasBaseClient.getClient(AtlasBaseClient.java:270)
    at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:453)
    at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:448)
    at org.apache.atlas.AtlasBaseClient.<init>(AtlasBaseClient.java:132)
    at org.apache.atlas.AtlasClientV2.<init>(AtlasClientV2.java:82)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:131)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more
Failed to import Hive Meta Data!!!

还是报错，提示找不到类，从server目录下拷贝到hook/hive目录下

# cp server/webapp/atlas/WEB-INF/lib/jackson-jaxrs-1.8.3.jar hook/hive/atlas-hive-plugin-impl/
# cp server/webapp/atlas/WEB-INF/lib/jackson-jaxrs-json-provider-2.9.2.jar hook/hive/atlas-hive-plugin-impl/
# cp server/webapp/atlas/WEB-INF/lib/jackson-module-jaxb-annotations-2.9.8.jar hook/hive/atlas-hive-plugin-impl/

再次执行成功，到atlas里可以看到hive相关数据

导入成功

原文地址：https://www.cnblogs.com/barneywill/p/12180277.html

时间： 2024-10-04 16:31:40

【原创】大叔经验分享（100）Atlas导入hive元数据的相关文章

【原创】大叔经验分享（23）hive metastore的几种部署方式

hive及其他组件(比如spark.impala等)都会依赖hive metastore,依赖的配置文件位于hive-site.xml hive metastore重要配置 hive.metastore.warehouse.dirhive2及之前版本默认为/user/hive/warehouse/,创建数据库或表时会在该目录下创建对应的目录 javax.jdo.option.ConnectionURLjavax.jdo.option.ConnectionDriverNamejavax.jdo.o

【原创】大叔经验分享（35）lzo格式支持

建表语句 CREATE EXTERNAL TABLE `my_lzo_table`(`something` string)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS INPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputForma

【原创】大叔经验分享（55）hue导出行数限制

/opt/cloudera/parcels/CDH/lib/hue/apps/beeswax/src/beeswax/conf.py # Deprecated DOWNLOAD_CELL_LIMIT = Config( key='download_cell_limit', default=10000000, type=int, help=_t('A limit to the number of cells (rows * columns) that can be downloaded from

【原创】大叔经验分享（71）docker容器中使用jvm工具

java应用中经常需要用到jvm工具来进行一些操作,如果java应用部署在docker容器中,如何使用jvm工具? 首先要看使用的docker镜像, 比如常用的openjdk镜像分为jdk和jre,只有jdk版本才有jvm工具,所以可以直接使用jdk版本的openjdk: 比如常用的tomcat镜像则没有jdk和jre选择,默认使用都是jre,所以没有jvm工具,tomcat镜像中的jdk目录如下: # ls /usr/lib/jvm/java-1.8-openjdk bin jre lib #

【原创】大叔经验分享（39）spark cache unpersist级联操作

问题:spark中如果有两个DataFrame(或者DataSet),DataFrameA依赖DataFrameB,并且两个DataFrame都进行了cache,将DataFrameB unpersist之后,DataFrameA的cache也会失效,官方解释如下: When invalidating a cache, we invalid other caches dependent on this cache to ensure cached data is up to date. For

【原创】大叔经验分享（52）ClouderaManager修改配置报错

Cloudera Manager中修改配置可能报错: Incorrect string value: '\xE7\xA8\x8B\xE5\xBA\x8F...' for column 'MESSAGE' at row 1 这是一个mysql的字符集问题,极有可能创建scm数据库时使用默认的latin1编码导致,涉及的表为: CREATE TABLE `REVISIONS` ( `REVISION_ID` bigint(20) NOT NULL, `OPTIMISTIC_LOCK_VERSION`

【原创】大叔经验分享（53）kudu报错unable to find SASL plugin: PLAIN

kudu安装后运行不正常,master中找不到任何tserver,查看tserver日志发现有很多报错: Failed to heartbeat to master:7051: Invalid argument: Failed to ping master at master:7051: Client connection negotiation failed: client connection to master:7051: unable to find SASL plugin: PLAIN

【原创】大叔经验分享（57）hue启动coordinator时报错

hue启动coordinator时报错,页面返回undefinied错误框: 后台日志报错: runcpserver.log [13/May/2019 04:34:55 -0700] middleware INFO Processing exception: 'NoneType' object has no attribute 'is_superuser': Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.

【原创】大叔经验分享（58）kudu写入压力大时报错

kudu写入压力大时报错 19/05/18 16:53:12 INFO AsyncKuduClient: Invalidating location fd52e4f930bc45458a8f29ed118785e3(server002:7050) for tablet 4259921cdcca4776b37771659a8cafb3: Service unavailable: Soft memory limit exceeded (at 106.05% of capacity). See htt