is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.6 GB of 40 GB virtual memory used

昨天使用hadoop跑五一的数据,发现报错:

Container [pid=47660,containerID=container_1453101066555_4130018_01_000067] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.6 GB of 40 GB virtual memory used. Killing container.

发现是内存溢出了,遇到这种问题首先要判断是map阶段溢出还是reduce阶段溢出,然后分别设置其内存的大小,比如:

在运行hive sql前加上 :  (map)
set mapreduce.map.memory.mb=2048
或者 (reduce)
set mapreduce.reduce.memory.mb=2048

因为默认值都是:1024M,也就是一个G,如果不够就会溢出!

时间: 2024-10-31 05:21:52

is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.6 GB of 40 GB virtual memory used的相关文章

spark运行任务报错:Container [...] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB physical memory used; 5.0 GB of 6.3 GB virtual memory used. Killing container.

spark版本:1.6.0 scala版本:2.10 报错日志: Application application_1562341921664_2123 failed 2 times due to AM Container for appattempt_1562341921664_2123_000002 exited with exitCode: -104 For more detailed output, check the application tracking page: http://w

Kafka:ZK+Kafka+Spark Streaming集群环境搭建(十三)kafka+spark streaming打包好的程序提交时提示虚拟内存不足(Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G)

异常问题:Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container. spark-submit提交脚本: [[email protected] work]$ more submit.sh #! /bin/bash jars=""

[hadoop] - Container [xxxx] is running beyond physical/virtual memory limits.

当运行mapreduce的时候,有时候会出现异常信息,提示物理内存或者虚拟内存超出限制,默认情况下:虚拟内存是物理内存的2.1倍.异常信息类似如下: Container [pid=13026,containerID=container_1449820132317_0013_01_000012] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.7 GB o

运行hadoop的时候提示物理内存或虚拟内存溢出的解决方案running beyond physical memory或者beyond vitual memory limits

当运行中出现Container is running beyond physical memory这个问题出现主要是因为物理内存不足导致的,在执行mapreduce的时候,每个map和reduce都有自己分配到内存的最大值,当map函数需要的内存大于这个值就会报这个错误,解决方法: 在mapreduc-site.xml配置里面设置mapreduce的内存分配大小 <property> <name>mapreduce.map.memory.mb</name> <va

【hadoop】 running beyond virtual memory错误原因及解决办法

问题描述: 在hadoop中运行应用,出现了running beyond virtual memory错误.提示如下: Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual

Spark- Spark Yarn模式下跑yarn-client无法初始化SparkConext,Over usage of virtual memory

在spark yarn模式下跑yarn-client时出现无法初始化SparkContext错误. 17/09/27 16:17:54 INFO mapreduce.Job: Task Id : attempt_1428293579539_0001_m_000003_0, Status : FAILED Container [pid=7847,containerID=container_1428293579539_0001_01_000005] is running beyond virtual

Memory Limits for Windows and Windows Server Releases

his topic describes the memory limits for supported Windows and Windows Server releases. Memory and Address Space Limits Physical Memory Limits: Windows 8 Physical Memory Limits: Windows Server 2012 Physical Memory Limits: Windows 7 Physical Memory L

ADDM Reports bug:Significant virtual memory paging was detected on the host operating system

查看ADDM(数据库版本为ORACLE 10.2.0.5.0)报告时,发现其中有个结论非常不靠谱:Significant virtual memory paging was detected on the host operating system,具体内容如下所示: FINDING 2: 100% impact (3930 seconds) ------------------------------------- Significant virtual memory paging was d

Extended paging tables to map guest physical memory addresses from virtual memory page tables to host physical memory addresses in a virtual machine system

A processor including a virtualization system of the processor with a memory virtualization support system to map a reference to guest-physical memory made by guest software executable on a virtual machine which in turn is executable on a host machin