排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space

报错栈:

2017-06-16 19:50:51,002 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: 1=1 AND 1=1
2017-06-16 19:50:51,043 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Working on split: 1=1 AND 1=1
2017-06-16 19:50:51,095 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "EXTEND3","EXTEND2","EXTEND1","MEMO","OPER_DATE","OPER_CODE","FILE_CONTENT","FILE_NAME","INPATIENT_NO","ID" from HIS_SDZL."MDT_FILE" tbl where ( 1=1 ) AND ( 1=1 )
2017-06-16 20:00:22,170 INFO [Thread-13] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
2017-06-16 20:00:22,185 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:514)
    at java.lang.StringBuffer.append(StringBuffer.java:352)
    at java.util.regex.Matcher.appendReplacement(Matcher.java:888)
    at java.util.regex.Matcher.replaceAll(Matcher.java:955)
    at java.lang.String.replaceAll(String.java:2223)
    at QueryResult.readFields(QueryResult.java:205)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

调小fetchsize参数也不能解决,那问题很可能是某行数据占用空间很大。根据Sqoop生成的导入表对应的实例化类QueryResult.java的244行可定位到报错列是FILE_CONTENT,是个二进制列, 然后查询原库,果然最大的列长达到180M:

ps: 怎么用标准的sql语句查询 blob字段的大小?
blob字段有好多种。如果是9i的简单的blob字段则应该是 length,或者lengthb也可。实在不行可以用 dbms_lob.getlength()

时间: 2024-10-23 08:24:37

排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space的相关文章

解决sqoop报错:java.lang.OutOfMemoryError: Java heap space

报错栈: 2017-06-15 16:24:50,449 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX6

【IntellJ IDEA】idea启动测试类报错Error running 'Test1.test': Command line is too long. Shorten command line for Test1.test or also for JUnit default configuration.

idea启动测试类报错 Error running 'Test1.test': Command line is too long. Shorten command line for Test1.test or also for JUnit default configuration. 注意: 最简单的方法,就是你重新创建一个新的测试类,在里面重新写一遍测试方法,代码都可以粘贴过去. 解决方法: 1.打开本项目的.idea文件夹,找到文件夹中的workspace.xml文件 2.搜索 Proper

Addnode.sh失败,日志文件报错PRKC-1025 & java.lang.OutOfMemoryError: Java heap space

环境: Linux 5.5 + RAC 11203 问题描述: 添加集群成员命令如下: $GRID_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={X3850AR}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={X3850AR-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={X3850AR-vip}" 报错如下: Instantiating

Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space

近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories to completehostConfig.deployWar=Deploying web application archive {0} java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap s

Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错

在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java -jar picard.jar SortVcf I=hg19.sites.vcf O=hg19.sites.sorted.vcf SEQUENCE_DICTIONARY=hg19.dict 出现这种报错说明内存不足.因此我们可以加上“-Xmx”参数 java -Xmx60g -jar picard.jar

eclipse中报错:java.lang.OutOfMemoryError: Java heap space

问题: 在eclipse中运行java程序,去重100多万的数据,报如下错误: java.lang.OutOfMemoryError: Java heap space 异常原因: 在JVM中如果98%的时间是用于GC且可用的 Heap size 不足2%的时候将抛出此异常信息. JVM堆的设置是指java程序运行过程中JVM可以调配使用的内存空间的设置.JVM在启动的时候会自动设置Heap size的值,其初始空间(即-Xms)是物理内存的1/64,最大空间(-Xmx)是物理内存的1/4.可以利

JDBC连接mysql,查大数据集报:java.lang.OutOfMemoryError: Java heap space

问题描述: 在项目中需要连接mysql,查询上千上万的记录,每个字段都挺大.结果在Eclipse中报如下错误: java.lang.OutOfMemoryError: Java heap space 原因分析: mysql会将查询到的记录全部发送到java端保存,而JVM中如果98%的时间是用于GC,且可用的Heap size 不足2%的时候将抛出此异常信息.JVM堆的设置是指java程序运行过程中JVM可以调配使用的内存空间的设置.JVM在启动的时候会自动设置Heap size的值,其初始空间

解决Scala异常处理java.lang.OutOfMemoryError: Java heap space error

需求:百万.千万.4千万级日志对设备进行除重环境:设备内存64G,scala单机版运行shell文件日志:20G 48000000.log4.0G 10000000.log396M 1000000.log 代码如下 帮助 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 import scala.io.Source import scala.collection.mutable.ArrayBuffer var te

关于java.lang.OutOfMemoryError: Java heap space的错误分析

今天无意间遇到这个错误:java.lang.OutOfMemoryError: Java heap space 问题出现原因:使用a标签实现快速下载[当然已经实现了,但想了想还是要归纳解决这类问题] <a href="/visecRepos/downFileServlet?filepath=/tdp/upload/14066209657…ename=%BC%AF%BF%D8%CF%B5%CD%B3%B0%B2%D7%B0%CE%C4%BC%FE_1.rar" target=&qu