Five Steps to Avoiding Java Heap Space Errors

来自:https://www.mapr.com/blog/how-to-avoid-java-heap-space-errors-understanding-and-managing-task-attempt-memory#.VMWvNDGUfXY

Keeping these five steps in mind can save you a lot of headaches and avoid Java heap space errors.

  1. Calculate memory needed.
  2. Check that the JVMs have enough memory for the TaskTracker tasks.
  3. Check that the JVMs settings are suitable for your tasks.
  4. Limit your nodes use of swap space and paged memory.
  5. Set the task attempt slots to a number that’s lower than the number calculated by the JobTracker web GUI.

In this blog, I will cover each of these steps which will give you a much better understanding and ability to manage task attempt memory.

It is important to understand how to manage task attempt memory, so that you can avoid Java heap space errors. When running map/reduce jobs, you may observe task attempts that fail like this:
 
13/09/20 08:50:56 INFO mapred.JobClient: Task Id : attempt_201309200652_0003_m_000000_0, Status : FAILED on node node1
 
Error: Java heap space
 
This error occurs when the task attempt tries to use more memory than the maximum limit set for the Java Virtual Machine (JVM) used to run it.
 
The first step in avoiding Java heap space errors is to understand memory requirements for your map and reduce tasks, so that you can start the JVM with an appropriate memory limit.
 
For example, in the wordcount job shipped in the hadoop-0.20.2-dev-examples.jar, map tasks would not need much memory, regardless of what data is being processed. The only significant amount of memory a map task would need would be for loading in libraries needed for execution. When using the default wordcount included with the MapR packages, 512MB of memory should be more than enough for the map attempt JVMs. If you plan to run the Hadoop examples we provide, you can look to spawn your map task attempt JVMs with a 512MB memory limit.
 
If you know how much memory your map task attempts should receive (in this case it’s 512MB), the next step is to launch the JVMs with that amount of memory. The task attempt JVM memory is set by the TaskTracker as it spawns JVMs to process data for a Map/Reduce job. The limit set by the TaskTracker comes from one of two possible sources: either the user specifies the desired amount of memory at the time they submit the job as part of their job conf object, or the TaskTracker will spawn the JVM with a default amount of memory.
 
The mapred.map.child.java.opts property is used to provide arguments that the TaskTracker uses to start JVMs to run map tasks (there is a similar property for reduce tasks). If themapred.map.child.java.opts property was set to "-Xmx512m" then the map task attempt JVMs would have a memory limit of 512MB. Alternately, when no -Xmx value is specified via the conf property, each TaskTracker will calculate a default memory limit for the JVMs that it launches. The limit will be based on the number of map/reduce task slots a TaskTracker will present, as well as the total Map/Reduce memory the TaskTracker has been instructed to use as a system-wide limit.
 
The number of map/reduce task attempt slots presented by a TaskTracker is set at the time the TaskTracker starts, and is controlled via two settings in mapred-site.xml on each node:
 
mapred.tasktracker.map.tasks.maximum mapred.tasktracker.reduce.tasks.maximum
 
The default values for these settings are formulas that are based on the number of CPU cores on the node. However, you can override the parameters by 1) modifying mapred-site.xml and setting a hard coded number of slots, or 2) using your own formula.
 
The system-wide TaskTracker map/reduce task attempt memory limit is set when a TaskTracker process starts. There are two possible sources for the memory limit that the TaskTracker will impose. First, a limit can be set explicitly in the hadoop-env.sh script in the hadoop conf directory. You can specify a memory limit by adding this line:
 
export HADOOP_HEAPSIZE=2000
 
This instructs the TaskTracker to limit the total memory that all task attempt JVMs on the node can use concurrently to 2000MB. If no HADOOP_HEAPSIZE parameter is specified in the hadoop-env.sh file, then the MapR warden service will provide the memory limit when it starts the TaskTracker. The warden service will calculate the limit it specifies based on the amount of physical memory on the node, minus the amount already committed to the services configured to run on the node. If you look in warden.conf you will see properties such as this:
 
service.command.mfs.heapsize.percent=20 service.command.mfs.heapsize.min=512
 
This particular example indicates the warden should instruct the MFS service to use 20% of the physical memory on the node or a minimum of 512MB (in the case that 512MB > 20% of physical memory). If you look at all the services configured to run on a node,as well as the memory specifications in warden.conf, you should be able to determine how much memory is being committed to the configured services (as well as how much is reserved for general OS/administration). The leftover memory is what is the TaskTracker willset as the memory limit for concurrently running task attempts.
 
For example, imagine that you have a single node install where you run ZooKeeper, CLDB, MFS, JobTracker, TaskTracker, NFS, the GUI, HBase Master and HBase RegionServer. That’s a lot of services on one node, and each one requires memory, so the warden will be allocating percentages of the memory to each service, chipping away at what’s left for map/reduce task attempts on the node. If you add up the percentages used by each of those services and it adds up to 60%, and there is a 5% OS reservation, you‘re left with 35% of the memory on the node for map/reduce task attempts. If you had 10GB of memory on the node, you would have 3.5GB for your map/reduce tasks. If you have 6 map slots and 4 reduce slots set on the node, then you would have 10 slots total.If the memory is divided evenly, you‘d end up with a JVM memory limit of 350MB for each JVM. If you need 512MB of memory to run your map tasks, then this default config won‘t work, and you will encounter Java heap space errors.
 
There are a few additional issues to be aware of when managing task attempt memory. Do not force your nodes to use any significant amount of swap space, or to cause memory to be paged into/out of disk frequently. If you change how you submit your job by explicitly setting "-Xmx500m" inmapred.map.child.java.opts, you‘ve overridden the safe memory limits, but you don‘t actually have additional physical memory. So while the map/reduce task attempts may launch, you may force massive amounts of swap to be used, and you can‘t rely on the kernel OOM killer or anything else to prevent this from occurring. You also can’t expect a node that starts paging heavily to recover quickly, if at all. If you just increase the task attempt JVM memory while continuing to launch the same number of task attempts concurrently on the node, you are subscribing more memory, and you need to be careful not to oversubscribe. If you are oversubscribing significantly,you might cause significant paging, and your nodes might freeze up and never come back online until power cycled.
 
So if you increase the amount of memory given to each task attempt JVM, you will also need to reduce the number of map/reduce task slots presented by TaskTrackers.
 
This is a complex situation, because if you run different jobs concurrently on a cluster, you can have task attempts from one job (jobA) that need lots of memory, and task attempts from another job (Job B) that need very little memory. Consequently, if you lower the number of map/reduce slots, you may find that you have plenty of free memory when a node is running tasks mostly for job B, but not that much free memory available for jobA. The key is to find the balance where you can allow some level of oversubscription, without being so aggressive that your nodes freeze.
 
To assist with this task, the TaskTracker will look at the amount of memory currently consumed by all running map/reduce tasks. It is not looking at the maximum limit on these tasks, but the actual amount of memory being utilized in total across all running task attempts. If that amount of memory reaches a specified level, the TaskTracker can kill some running task attempts to free up memory so that the other task attempts can finish successfully without causing a node to start paging excessively.
 
For example, if you’re trying to get the wordcount example to run on a small cluster or single node setup, and it fails with a "Java heap space" error, the easiest, fastest solution is to edit/opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml on the TaskTracker node(s) and to reduce the number of map/reduce task attempt slots by setting:
 
mapred.tasktracker.map.tasks.maximum mapred.tasktracker.reduce.tasks.maximum
 
It’s important to set the task attempt slots to a number that’s lower than what’s being calculated currently, which can be determined by going to the JobTracker web GUI. For example, if you had one TaskTracker and it was showing 6 map slots and 4 reduce slots, then you should set that number to 3 map slots and 2 reduce slots,and restart the TaskTracker process on the node via:
 
maprcli node services -nodes -tasktracker restart
 
When it restarts with the lowered number of slots, re-submit the wordcount job. Witheach attempt, JVM should be getting more memory without any additional oversubscription. It’s a safe solution, as the nodes shouldn‘t start paging aggressively, it’s an easy solution since it doesn’t require a lot of memory calculations, and it’s fast since the solution requires an edit to just one config file and a service restart command afterwards.
In order to avoid Java heap space errors, keep the following points in mind:

  1. Calculate how much memory your task attempts will need.
  2. Make sure that the TaskTracker launches your task attempts in JVMs that have a memory limit greater than or equal to your expected memory requirement.
  3. Remember that there are default settings that come into play when launching those JVMs, unless you have explicitly overridden them. The default settings may not be suitable based on the balance of CPU cores, physical memory and the set of services that you have configured to run on the node.
  4. Do not force your nodes to use any significant amount of swap space, or to cause memory to be paged into/out of disk frequently.
  5. Set the task attempt slots to a number that’s lower than what is being calculated by the JobTracker web GUI.
时间: 2024-10-10 16:04:38

Five Steps to Avoiding Java Heap Space Errors的相关文章

Tomcat 优化 java.lang.OutOfMemoryError: Java heap space 的解决方法

java.lang.OutOfMemoryError: Java heap space 的解决方法 关键字: tomcat outofmemoryerror permgen space java heap space 最近在熟悉一个开发了有几年的项目,需要把数据库从mysql移植到oracle,首先把jdbc的连接指向mysql,打包放到tomcat里面,可以跑起来,没有问题,可是当把jdbc连接指向oracle的时候,tomcat就连续抛java.lang.OutOfMemoryError的错

java.lang.OutOfMemoryError: Java heap space解决办法

//首先检查程序有没有限入死循环 这个问题主要还是由这个问题 java.lang.OutOfMemoryError: Java heap space 引起的.第一次出现这样的的问题以后,引发了其他的问题.在网上一查可能是JAVA的堆栈设置太小的原因. 跟据网上的答案大致有这两种解决方法: 1.设置环境变量 解决方法:手动设置Heap size 修改TOMCAT_HOME/bin/catalina.sh set JAVA_OPTS= -Xms32m -Xmx512m 可以根据自己机器的内存进行更改

从[java.lang.OutOfMemoryError: Java heap space]中恢复

出现java.lang.OutOfMemoryError: Java heap space的错误要么是程序问题,要么就是分配给JVM的内存确实不够. 一般来说都是可以事前可控解决的. 但是如果不可控的情况,例如使用第三方包,或者系统抽筋,就会抛出OutOfMemoryError错误.OH NO,根据不会抛出来,当前线程直接挂掉. 既然都挂掉了,谈什么恢复?而且挂掉也不只是OutOfMemoryError的问题. 一般情况下,OutOfMemoryError在不可控的情况下,真的真的真的不需要处理

Eclipse运行程序提示:Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

问题描述: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space 问题原因: 程序中对象引用过多导致堆空间不足,导致内存溢出 解决方案: (增大Java虚拟机的内存空间) 打开Eclipse,选择"Run" - "Run Configurations" - "(x)=Arguments",VM arguments栏中填写 -Xmx800m

解决sqoop报错:java.lang.OutOfMemoryError: Java heap space

报错栈: 2017-06-15 16:24:50,449 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX6

内存大小设置 Java heap space错误(finereport报表)

1. 问题描述 当从数据库中查询大量的数据,每个模板取出来几百万条数据,或者是频繁的刷新项目.模板时就会占用Java虚拟机JVM的大量内存,超过内存就会出现报java.lang.OutOfMemoryError:Java heap space内存一处的错误,具体报错如下: 2. 原因 由于服务器的JVM不够用而抛出的错误,JVM在启动的时候会自动设置Heap size的值,初始空间(即-Xms)是物理内存的1/64,最大空间(-Xmx)是物理内存的1/4.所以可以根据自己的情况进行修改JVM的-

java.lang.OutOfMemoryError: PermGen space PermGen space & java.lang.OutOfMemoryError: Java heap space Heap siz

java.lang.OutOfMemoryError: PermGen space PermGen space 由-XX:PermSize  -XX:MaxPermSize 引起 java.lang.OutOfMemoryError: Java heap space Heap siz 由-Xms -Xmx 引起 Liunx下修改:catalina.sh # OS specific support.  $var _must_ be set to either true or false. JAVA

Hadoop on Mac with IntelliJ IDEA - 5 解决java heap space问题

本文讲述在CentOS 6.5中提交作业到hadoop 1.2.1于reduce阶段遇到Error: java heap space错误导致作业重新计算的解决过程.解决办法适用Linux.Mac OS X 和Windows操作系统. 环境:Mac OS X 10.9.5, IntelliJ IDEA 13.1.4, Hadoop 1.2.1 Hadoop放在虚拟机中,宿主机通过SSH连接,IDE和数据文件在宿主机.IDEA自身运行于JDK 1.8,IDEA工程及Hadoop使用JDK 1.6.

Exception in thread "HSQLDB Connection @429be9" java.lang.OutOfMemoryError: Java heap space

Exception in thread "HSQLDB Connection @429be9" java.lang.OutOfMemoryError: Java heap space 在服务器下的data/xxxx也就是数据库路径下,我的在F:\tomcat7\bin\data下删掉关于xxxx的文件及文件夹. Exception in thread "HSQLDB Connection @429be9" java.lang.OutOfMemoryError: Ja