YARN Resource Management

?Chapter 11. Using the YARN CLI to View Logs for Running Applications

This chapter describes how to use the YARN CLI (Command Line Interface) to view log files for running applications. You can access container log files using the YARN ResourceManager web UI, but more options are available when you use the yarn logs CLI command.

?View all Log Files for a Running Application

Use the following command format to view all logs for a running application:

yarn logs -applicationId <Application ID>

?View a Specific Log Type for a Running Application

Use the following command format to view all logs of a particular type for a running application:

yarn logs -applicationId <Application ID> -log_files <log_file_type>

For example, to view only the stderr error logs:

yarn logs -applicationId <Application ID> -log_files stderr

The -logFiles option also supports Java regular expressions, so the following format would return all types of log files:

yarn logs -applicationId <Application ID> -log_files .* 

?View ApplicationMaster Log Files

Use the following command format to view all ApplicationMaster container log files for a running application:

yarn logs -applicationId <Application ID> -am ALL

Use the following command format to view only the first ApplicationMaster containter log files:

yarn logs -applicationId <Application ID> -am 1

?List Container IDs

Use the following command format to list all container IDs for a running application:

yarn logs -applicationId <Application ID> -show_application_log_info

?View Log Files for One Container

Once you have the container IDs, you can use the following command format to list the log files for a particular container:

yarn logs -applicationId <Application ID> -containerId <Container ID>

?Show Container Log File Information

Use the following command format to list all of the container log file names (types) for a running application:

yarn logs -applicationId <Application ID> -show_container_log_info

You can then use the -logFiles option to view a particular log type.

?View a Portion of the Log Files for One Container

For large container log files, you can use the following command format to list only a portion of the log files for a particular container:

yarn logs -applicationId <Application ID> -containerId <Container ID> -size <bytes>

To view the first 1000 bytes:

yarn logs -applicationId <Application ID> -containerId <Container ID> -size 1000

To view the last 1000 bytes:

yarn logs -applicationId <Application ID> -containerId <Container ID> -size -1000

?Download Logs for a Running Application

Use the following command format to download logs to a local folder:

yarn logs -applicationId <Application ID> -out <path_to_local_folder>

The container log files are organized in parent folders labeled with the applicable node ID.

?Display Help for YARN Logs

To display Help for yarn logs, run the following commmand:

yarn logs -help

Commands for testing

These operations are here primarily for testing.

kill-container <name> --id container-id

Kill a YARN container belong to the application. This is useful primarily for testing the resilience to failures.

Container IDs can be determined from the application instance status JSON document.

原文地址:https://www.cnblogs.com/WCFGROUP/p/10317496.html

时间: 2024-10-17 03:40:39

YARN Resource Management的相关文章

【IOS笔记】Resource Management in View Controllers

Resource Management in View Controllers 视图控制器的资源管理 View controllers are an essential part of managing your app’s resources. View controllers allow you to break your app up into multiple parts and instantiate only the parts that are needed. But more t

Hadoop 管理工具HUE配置-Yarn Resource Manager HA配置

安装HUE之后,需要配置很多东西才能将这个系统的功能发挥出来,因为Yarn是配置的HA模式,所以在配置HUE的时候,会有些不用,下面一段文字是官网拿来的 # Configuration for YARN (MR2) # ------------------------------------------------------------------------ [[yarn_clusters]] [[[default]]] # Whether to submit jobs to this cl

&lt;Effective C++&gt;读书摘要--Resource Management&lt;一&gt;

1.除了内存资源以外,Other common resources include file descriptors, mutex locks, fonts and brushes in graphical user interfaces (GUIs), database connections, and network sockets. Regardless of the resource, it's important that it be released when you're fini

Copy Control and Resource Management

Ordinarily classes that manage resources that do not reside in the class must define the copy-control members. Once a class needs a destructor, it almost surely needs a copy constructor and copy-assignment operator as well. In order to define these m

device resource management

http://www.wowotech.net/device_model/device_resource_management.html

Containers Reserved yarn resourcemanager

yarn rm的管理页面中显示了集群的概况,其中有一个指标叫Containers Reserved . 预留的容器,为什么会预留,集群的资源使用饱合,新的app请求的资源一般会进入pending状态,为什么需要预留, 查阅资料说是,如果app申请的资源不太容易分配,比如新的app是一个计算密集型的,一个task需要6个vcores,其他任务需要一个, 则刚释放的资源如果不预留,就会被其他任务使用,而这样有失公平. 这也是yarn 公平调度器的一个特性,有关公平调度器的更多说明,请阅读: Fair

spark 笔记 4:Apache Hadoop YARN: Yet Another Resource Negotiator

spark支持YARN做资源调度器,所以YARN的原理还是应该知道的:http://www.socc2013.org/home/program/a5-vavilapalli.pdf    但总体来说,这是一篇写得一般的论文,它的原理没有什么特别突出的,而且它列举的数据没有对比性,几乎看不出YARN有什么优势.反正我看完的感觉是,YARN的资源分配在延迟上估计很糟糕.而实际使用似乎也印证了这个预感. Abstract  two key shortcomings: 1) tight coupling

# Apache Hadoop Yarn: Yet Another Resource Negotiator论文解读

纯属云平台管理学习菜鸟的笔记,参照许多大牛的博客,如有侵权,请联系,立刻删除. Abstract 1) tight coupling of a specific programming model with the re- source management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' contro

Docker on YARN在Hulu的实现

这篇文章是我来Hulu这一年做的主要工作,结合当下流行的两个开源方案Docker和YARN,提供了一套灵活的编程模型,目前支持DAG编程模型,将会支持长服务编程模型. 基于Voidbox,开发者可以很容易的写出一个分布式的框架,Docker作为运行的执行引擎,YARN作为集群资源的管理系统. 同时这篇文章也发表在Hulu官方的技术博客上:http://tech.hulu.com/blog/2015/08/06/voidbox-docker-on-yarn/ 1. Voidbox Motivati