解决sqoop报错：java.lang.OutOfMemoryError: Java heap space

报错栈：

2017-06-15 16:24:50,449 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX67","CTJX68","CTJX69","CTJX70","CTJX71","CTJX72","CTJX73","CTJX74","CTJX75","CTJX76","CTJX77","CTJX78","CTJX79","CTJX80","CTJX81","CTJX82","CTJX83","CTJX84","CTJX85","CTJX86","CTJX87","CTJX88","CTJX89","CTJX90","CTJX91","CTJX92","CTJX93","CTJX94","CTJX95","CTJX96","CTJX97","CTJX98","CTJX99","CTJX55","IID","IEXAM_IID","CTSBGLX","CTJX01","CTJX02","CTJX03","CTJX04","CTJX05","CTJX06","CTJX07","CTJX08","CTJX09","CTJX10","CTJX11","CTJX12","CTJX13","CTJX14","CTJX15","CTJX16","CTJX17","CTJX18","CTJX19","CTJX20","CTJX21","CTJX22","CTJX23","CTJX24","CTJX25","CTJX26","CTJX27","CTJX28","CTJX29","CTJX30","CTJX31","CTJX32","CTJX33","CTJX34","CTJX35","CTJX36","CTJX37","CTJX38","CTJX39","CTJX40","CTJX41","CTJX42","CTJX43","CTJX44","CTJX45","CTJX46","CTJX47","CTJX48","CTJX49","CTJX50","CTJX51","CTJX52","CTJX53","CTJX54","CTJX56","CTJX57","CTJX58","CTJX59" from RIS."ICNRIS_UIS_EXAM_TJX" tbl where ( 1=1 ) AND ( 1=1 )
2017-06-15 16:24:50,589 INFO [Thread-13] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
2017-06-15 16:24:50,604 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:870)
    at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1047)
    at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:850)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1134)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3384)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

解决：调小sqoop参数：--fetch-size

解决过程，查看sqoop源码，看到fetchSize，想到调整这个参数：

ps: 调大mapper参数不管用，曾设置过参数：-D mapreduce.map.memory.mb=8192 -D yarn.app.mapreduce.am.resource.mb=6144。问题是在从db fetch数据的时候出现的，原表的列不算多，但每个列的长度很大：

时间： 2024-12-15 05:48:04

解决sqoop报错：java.lang.OutOfMemoryError: Java heap space的相关文章

排查sqoop报错：Error running child : java.lang.OutOfMemoryError: Java heap space

报错栈: 2017-06-16 19:50:51,002 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: 1=1 AND 1=1 2017-06-16 19:50:51,043 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Working on split: 1=1 AND 1=1 2017-06-16 19:50:51,095 INFO [mai

Addnode.sh失败,日志文件报错PRKC-1025 & java.lang.OutOfMemoryError: Java heap space

环境: Linux 5.5 + RAC 11203 问题描述: 添加集群成员命令如下: $GRID_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={X3850AR}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={X3850AR-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={X3850AR-vip}" 报错如下: Instantiating

Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space

近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories to completehostConfig.deployWar=Deploying web application archive {0} java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap s

Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错

在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java -jar picard.jar SortVcf I=hg19.sites.vcf O=hg19.sites.sorted.vcf SEQUENCE_DICTIONARY=hg19.dict 出现这种报错说明内存不足.因此我们可以加上“-Xmx”参数 java -Xmx60g -jar picard.jar

解决eclipse maven install 造成JVM 内存溢出(java.lang.OutOfMemoryError: Java heap space)

maven install 报错信息: The system is out of resources.Consult the following stack trace for details.java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.createEntry(HashMap.java:897) at java.util.HashMap.addEntry(HashMap.java:884) at java.ut

java.lang.OutOfMemoryError: Java heap space解决办法

//首先检查程序有没有限入死循环 这个问题主要还是由这个问题 java.lang.OutOfMemoryError: Java heap space 引起的.第一次出现这样的的问题以后,引发了其他的问题.在网上一查可能是JAVA的堆栈设置太小的原因. 跟据网上的答案大致有这两种解决方法: 1.设置环境变量 解决方法:手动设置Heap size 修改TOMCAT_HOME/bin/catalina.sh set JAVA_OPTS= -Xms32m -Xmx512m 可以根据自己机器的内存进行更改

java.lang.OutOfMemoryError: Java heap space解决方法

//首先检查程序有没有限入死循环 这个问题主要还是由这个问题 java.lang.OutOfMemoryError: Java heap space 引起的.第一次出现这样的的问题以后,引发了其他的问题.在网上一查可能是JAVA的堆栈设置太小的原因. 跟据网上的答案大致有这两种解决方法: 1.设置环境变量 解决方法:手动设置Heap size 修改TOMCAT_HOME/bin/catalina.sh set JAVA_OPTS= -Xms32m -Xmx512m 可以根据自己机器的内存进行更改

Tomcat 优化 java.lang.OutOfMemoryError: Java heap space 的解决方法

java.lang.OutOfMemoryError: Java heap space 的解决方法 关键字: tomcat outofmemoryerror permgen space java heap space 最近在熟悉一个开发了有几年的项目,需要把数据库从mysql移植到oracle,首先把jdbc的连接指向mysql,打包放到tomcat里面,可以跑起来,没有问题,可是当把jdbc连接指向oracle的时候,tomcat就连续抛java.lang.OutOfMemoryError的错

转载:java.lang.OutOfMemoryError: Java heap space解决方法

//首先检查程序有没有限入死循环 这个问题主要还是由这个问题 java.lang.OutOfMemoryError: Java heap space 引起的.第一次出现这样的的问题以后,引发了其他的问题.在网上一查可能是JAVA的堆栈设置太小的原因.跟据网上的答案大致有这两种解决方法:1.设置环境变量 解决方法:手动设置Heap size修改TOMCAT_HOME/bin/catalina.shset JAVA_OPTS= -Xms32m -Xmx512m可以根据自己机器的内存进行更改. 2.j