【hadoop】 running beyond virtual memory错误原因及解决办法

问题描述:

在hadoop中运行应用,出现了running beyond virtual memory错误。提示如下:

Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

原因:从机上运行的Container试图使用过多的内存,而被NodeManager kill掉了。
[摘录] The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don‘t want.
解决方法:mapred-site.xml中设置map和reduce任务的内存配置如下:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>1536</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx1024M</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>3072</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx2560M</value>
</property>

附录:

Container is running beyond memory limits

http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

时间: 2024-10-11 07:41:23

【hadoop】 running beyond virtual memory错误原因及解决办法的相关文章

[转]PLS-S-00201, identifier &#39;CALLDEMO.GET_EMPLOYEES&#39; must be declared 预编译错误原因及解决办法

$ proc sample9.pc SQLCHECK=SEMANTICS Pro*C/C++: Release 11.2.0.1.0 - Production on Tue Jan 8 15:18:40 2013 Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved. System default option values taken from: /usr/local/oracle/produc

Unknown system variable &#39;query_cache_size&#39; 错误原因以及解决办法

Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Unknown system variable

ORA-00845错误原因及解决办法

故障现象: 用startup命令启动数据库时出现ora-00845错误提示: SQL> startup ORA-00845: MEMORY_TARGET not supported on this system 故障原因: 在oracle database 11g中新增的内存自动管理的参数MEMORY_TARGET,它能自动调整SGA和PGA,这个特性需要用到/dev/shm共享文件系统,而且要求/dev/shm必须大于MEMORY_TARGET,如果/dev/shm比MEMORY_TARGET

服务器出现 HTTP 400、404、500、502 错误原因以及解决办法

一.基本包含了所有问题 HTTP 400 – 请求无效. HTTP 401.1 – 未授权:登录失败. HTTP 401.2 – 未授权:服务器配置问题导致登录失败. HTTP 401.3 – ACL 禁止访问资源. HTTP 401.4 – 未授权:授权被筛选器拒绝. HTTP 401.5 – 未授权:ISAPI 或 CGI 授权失败. 1.HTTP 403 – 禁止访问 HTTP 403 – 对 Internet 服务管理器 的访问仅限于 Localhost. HTTP 403.1 禁止访问

matlab 中出现的错误 原因及解决办法

1.Indexing cannot yield multiple results 原因:变量名与内置的函数名相同,比如:[min, k] = min(C) 解决办法:更改变量名:[m, k] = min(C)

You must supply a layout_width attribute的错误原因及解决办法

学习android的过程中,尝试新功能,结果出现了这个一个error: 05-21 15:38:52.745: E/AndroidRuntime(17608): java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.ex1/com.example.ex1.AnotherActivity}: java.lang.RuntimeException: Binary XML file line #6:

spring项目报org.apache.tiles.definition.DefinitionsFactoryException: I/O错误原因及解决办法。

今天升级一个spring项目遇到如下错: HTTP Status 500 - Request processing failed; nested exception is org.apache.tiles.definition.DefinitionsFactoryException: I/O Error reading definitions. org.springframework.web.util.NestedServletException: Request processing fail

Kafka:ZK+Kafka+Spark Streaming集群环境搭建(十三)kafka+spark streaming打包好的程序提交时提示虚拟内存不足(Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G)

异常问题:Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container. spark-submit提交脚本: [[email protected] work]$ more submit.sh #! /bin/bash jars=""

coreseek常见错误原因及解决方法

coreseek常见错误原因及解决方法 Coreseek 中文全文检索引擎 Coreseek 是一款中文全文检索/搜索软件,以GPLv2许可协议开源发布,基于Sphinx研发并独立发布,专攻中文搜索和信息处理领域,适用于行业/垂直搜索.论坛/站内搜索.数据库搜索.文档/文献检索.信息检索.数据挖掘等应用场景,用户可以免费下载使用 本文为大家整理了coreseek/sphinx中文检索引擎的常见问题和解决方法,感兴趣的同学参考下. Coreseek 是一款中文全文检索/搜索软件,以GPLv2许可协