hive:默认允许动态分区个数为100,超出抛出异常:

在创建好一个分区表后,执行动态分区插入数据,抛出了错误:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by
hive.exec.max.dynamic.partitions and
hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
 2017-11-06 14:54:31,381 FATAL [IPC Server handler 9 on 27102] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1500969698103_8078701_m_000278_0 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"gridid":"148420952","longitude":"120.315241543934","latitude":"30.191598257425056","objectid":"888665533","cellname":"7777建设_1","scrsrp":"-91.67163276672363","gridx":"12001","gridy":"7825","doortype":null,"biuldingid":null,"calibrategridid":"309","nobjectid":"193619203","ncrsrp":"-122.38408386707306","height":"0","p_group":0}
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:180)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:174)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"gridid":"148420952","longitude":"120.315241543934","latitude":"30.191598257425056","objectid":"998867665","cellname":"777建设_1","scrsrp":"-91.67163276672363","gridx":"12001","gridy":"7825","doortype":null,"biuldingid":null,"calibrategridid":"309","nobjectid":"193619203","ncrsrp":"-122.38408386707306","height":"0","p_group":0}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:936)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:713)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:111)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
    at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
    ... 9 more

解决方案:

在执行插入数据到分区时,添加参数设置:

set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions.pernode=10000;
set hive.exec.max.dynamic.partitions=10000;
set hive.exec.max.created.files=10000;
时间: 2024-11-05 22:47:48

hive:默认允许动态分区个数为100,超出抛出异常:的相关文章

Hive学习之动态分区及HQL

Hive动态分区 1.首先创建一个分区表create table t10(name string) partitioned by(dt string,value string)row format delimited fields terminatedby '\t' lines terminated by '\n'stored as textfile;2.然后对hive进行设置,使之支持动态分区,set hive.exec.dynamic.partition.mode=nonstrict;如果限

Hive动态分区

Hive默认是静态分区,我们在插入数据的时候要手动设置分区,如果源数据量很大的时候,那么针对一个分区就要写一个insert,比如说,我们有很多日志数据,我们要按日期作为分区字段,在插入数据的时候我们不可能手动的去添加分区,那样太麻烦了.还好,Hive提供了动态分区,动态分区简化了我们插入数据时的繁琐操作. 使用动态分区的时候必须开启动态分区(动态分区默认是关闭的),语句如下: [java] view plain copy set hive.exec.hynamic.partition=true;

hive从查询中获取数据插入到表或动态分区

(前人写的不错,很实用,负责任转发)转自:http://www.crazyant.net/1197.html Hive的insert语句能够从查询语句中获取数据,并同时将数据Load到目标表中.现在假定有一个已有数据的表staged_employees(雇员信息全量表),所属国家cnty和所属州st是该表的两个属性,我们做个试验将该表中的数据查询出来插入到另一个表employees中. 1 2 3 4 INSERT OVERWRITE TABLE employees PARTITION (cou

HIVE动态分区实战

一)hive中支持两种类型的分区: 静态分区SP(static partition) 动态分区DP(dynamic partition) 静态分区与动态分区的主要区别在于静态分区是手动指定,而动态分区是通过数据来进行判断.详细来说,静态分区的列实在编译时期,通过用户传递来决定的:动态分区只有在SQL执行时才能决定. 二)实战演示如何在hive中使用动态分区 1.创建一张分区表,包含两个分区dt和ht表示日期和小时 CREATE TABLE partition_table001 ( name ST

hive 动态分区

非常重要的动态分区属性: hive.exec.dynamic.partition  是否启动动态分区.false(不开启) true(开启)默认是 false hive.exec.dynamic.partition.mode  打开动态分区后,动态分区的模式,有 strict和 nonstrict 两个值可选,strict 要求至少包含一个静态分区列,nonstrict则无此要求.各自的好处,大家自己查看哈. hive.exec.max.dynamic.partitions 允许的最大的动态分区

大数据--hive动态分区调整

1.创建一张普通表加载数据 ------------------------------------------------ hive (default)> create table person(id int,name string,location string) > row format delimited fields terminated by '\t';OKTime taken: 0.415 seconds -------------------------------------

Hive分区(静态分区+动态分区)

Hive分区的概念与传统关系型数据库分区不同. 传统数据库的分区方式:就oracle而言,分区独立存在于段里,里面存储真实的数据,在数据进行插入的时候自动分配分区. Hive的分区方式:由于Hive实际是存储在HDFS上的抽象,Hive的一个分区名对应一个目录名,子分区名就是子目录名,并不是一个实际字段. 所以可以这样理解,当我们在插入数据的时候指定分区,其实就是新建一个目录或者子目录,或者在原有的目录上添加数据文件. Hive分区的创建 Hive分区是在创建表的时候用Partitioned b

Hive的静态分区和动态分区

作者:Syn良子 出处:http://www.cnblogs.com/cssdongl/p/6831884.html 转载请注明出处 虽然之前已经用过很多次hive的分区表,但是还是找时间快速回顾总结一下加深理解. 举个栗子,基本需求就是Hive有一张非常详细的原子数据表original_device_open,而且还在不断随着时间增长,那么我需要给它进行分区,为什么要分区?因为我想缩小查询范围,提高速度和性能. 分区其实是物理上对hdfs不同目录进行数据的load操作,0.7之后的版本都会自动

Hive Experiment 2(表动态分区和IDE)

1.使用oracle sql developer 4.0.3作为hive query的IDE. 下载hive-jdbc driver http://www.cloudera.com/content/cloudera/en/downloads/connectors/hive/jdbc/hive-jdbc-v2-5-6.html Start    Oracle    SQL    Developer    and    navigate    to    Preferences    |    Da