hadoop集群之HDFS和YARN启动和停止命令

假如我们只有3台linux虚拟机,主机名分别为hadoop01、hadoop02和hadoop03,在这3台机器上,hadoop集群的部署情况如下:

hadoop01:1个namenode,1个datanode,1个journalnode,1个zkfc,1个resourcemanager,1个nodemanager;

hadoop02:1个namenode,1个datanode,1个journalnode,1个zkfc,1个resourcemanager,1个nodemanager;

hadoop03:1个datenode,1个journalnode,1个nodemanager;

下面我们来介绍启动hdfs和yarn的一些命令。

1.启动hdfs集群(使用hadoop的批量启动脚本)

/root/apps/hadoop/sbin/start-dfs.sh

[[email protected] ~]# /root/apps/hadoop/sbin/start-dfs.sh
Starting namenodes on [hadoop01 hadoop02]
hadoop01: starting namenode, logging to /root/apps/hadoop/logs/hadoop-root-namenode-hadoop01.out
hadoop02: starting namenode, logging to /root/apps/hadoop/logs/hadoop-root-namenode-hadoop02.out
hadoop03: starting datanode, logging to /root/apps/hadoop/logs/hadoop-root-datanode-hadoop03.out
hadoop02: starting datanode, logging to /root/apps/hadoop/logs/hadoop-root-datanode-hadoop02.out
hadoop01: starting datanode, logging to /root/apps/hadoop/logs/hadoop-root-datanode-hadoop01.out
Starting journal nodes [hadoop01 hadoop02 hadoop03]
hadoop03: starting journalnode, logging to /root/apps/hadoop/logs/hadoop-root-journalnode-hadoop03.out
hadoop02: starting journalnode, logging to /root/apps/hadoop/logs/hadoop-root-journalnode-hadoop02.out
hadoop01: starting journalnode, logging to /root/apps/hadoop/logs/hadoop-root-journalnode-hadoop01.out
Starting ZK Failover Controllers on NN hosts [hadoop01 hadoop02]
hadoop01: starting zkfc, logging to /root/apps/hadoop/logs/hadoop-root-zkfc-hadoop01.out
hadoop02: starting zkfc, logging to /root/apps/hadoop/logs/hadoop-root-zkfc-hadoop02.out
[[email protected] ~]# 

从上面的启动日志可以看出,start-dfs.sh这个启动脚本是通过ssh对多个节点的namenode、datanode、journalnode以及zkfc进程进行批量启动的。

2.停止hdfs集群(使用hadoop的批量启动脚本)

/root/apps/hadoop/sbin/stop-dfs.sh 

[[email protected] ~]# /root/apps/hadoop/sbin/stop-dfs.sh
Stopping namenodes on [hadoop01 hadoop02]
hadoop02: stopping namenode
hadoop01: stopping namenode
hadoop02: stopping datanode
hadoop03: stopping datanode
hadoop01: stopping datanode
Stopping journal nodes [hadoop01 hadoop02 hadoop03]
hadoop03: stopping journalnode
hadoop02: stopping journalnode
hadoop01: stopping journalnode
Stopping ZK Failover Controllers on NN hosts [hadoop01 hadoop02]
hadoop01: stopping zkfc
hadoop02: stopping zkfc
[[email protected] ~]# 

3.启动单个进程

[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /root/apps/hadoop/logs/hadoop-root-namenode-hadoop01.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /root/apps/hadoop/logs/hadoop-root-namenode-hadoop02.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /root/apps/hadoop/logs/hadoop-root-datanode-hadoop01.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /root/apps/hadoop/logs/hadoop-root-datanode-hadoop02.out
[[email protected] apps]# /root/apps/hadoop/sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /root/apps/hadoop/logs/hadoop-root-datanode-hadoop03.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /root/apps/hadoop/logs/hadoop-root-journalnode-hadoop01.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /root/apps/hadoop/logs/hadoop-root-journalnode-hadoop02.out
[[email protected] apps]# /root/apps/hadoop/sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /root/apps/hadoop/logs/hadoop-root-journalnode-hadoop03.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start zkfc
starting zkfc, logging to /root/apps/hadoop/logs/hadoop-root-zkfc-hadoop01.out
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh start zkfc
starting zkfc, logging to /root/apps/hadoop/logs/hadoop-root-zkfc-hadoop02.out

分别查看启动后3台虚拟机上的进程情况:

[[email protected] ~]# jps
6695 DataNode
2002 QuorumPeerMain
6879 DFSZKFailoverController
7035 Jps
6800 JournalNode
6580 NameNode
[[email protected] ~]# 

[[email protected] ~]# jps
6360 JournalNode
6436 DFSZKFailoverController
2130 QuorumPeerMain
6541 Jps
6255 DataNode
6155 NameNode
[[email protected] ~]# 

[[email protected] apps]# jps
5331 Jps
5103 DataNode
5204 JournalNode
2258 QuorumPeerMain
[[email protected] apps]# 

3.停止单个进程

[[email protected] ~]# jps
6695 DataNode
2002 QuorumPeerMain
8486 Jps
6879 DFSZKFailoverController
6800 JournalNode
6580 NameNode
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop zkfc
stopping zkfc
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop journalnode
stopping journalnode
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop datanode
stopping datanode
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop namenode
stopping namenode
[[email protected] ~]# jps
2002 QuorumPeerMain
8572 Jps
[[email protected] ~]# 

[[email protected] ~]# jps
6360 JournalNode
6436 DFSZKFailoverController
2130 QuorumPeerMain
7378 Jps
6255 DataNode
6155 NameNode
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop zkfc
stopping zkfc
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop journalnode
stopping journalnode
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop datanode
stopping datanode
[[email protected] ~]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop namenode
stopping namenode
[[email protected] ~]# jps
7455 Jps
2130 QuorumPeerMain
[[email protected] ~]# 

[[email protected] apps]# jps
5103 DataNode
5204 JournalNode
5774 Jps
2258 QuorumPeerMain
[[email protected] apps]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop journalnode
stopping journalnode
[[email protected] apps]# /root/apps/hadoop/sbin/hadoop-daemon.sh stop datanode
stopping datanode
[[email protected] apps]# jps
5818 Jps
2258 QuorumPeerMain
[[email protected] apps]# 

3.启动yarn集群(使用hadoop的批量启动脚本)

/root/apps/hadoop/sbin/start-yarn.sh 

[[email protected] ~]# /root/apps/hadoop/sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /root/apps/hadoop/logs/yarn-root-resourcemanager-hadoop01.out
hadoop03: starting nodemanager, logging to /root/apps/hadoop/logs/yarn-root-nodemanager-hadoop03.out
hadoop02: starting nodemanager, logging to /root/apps/hadoop/logs/yarn-root-nodemanager-hadoop02.out
hadoop01: starting nodemanager, logging to /root/apps/hadoop/logs/yarn-root-nodemanager-hadoop01.out
[[email protected] ~]# 

从上面的启动日志可以看出,start-yarn.sh启动脚本只在本地启动一个ResourceManager进程,而3台机器上的nodemanager都是通过ssh的方式启动的。所以hadoop02机器上的ResourceManager需要我们手动去启动。

4.启动hadoop02上的ResourceManager进程

/root/apps/hadoop/sbin/yarn-daemon.sh start resourcemanager

5.停止yarn

/root/apps/hadoop/sbin/stop-yarn.sh

[[email protected] ~]# /root/apps/hadoop/sbin/stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
hadoop01: stopping nodemanager
hadoop03: stopping nodemanager
hadoop02: stopping nodemanager
no proxyserver to stop
[[email protected] ~]# 

通过上面的停止日志可以看出,stop-yarn.sh脚本只停止了本地的那个ResourceManager进程,所以hadoop02上的那个resourcemanager我们需要单独去停止。

6.停止hadoop02上的resourcemanager

/root/apps/hadoop/sbin/yarn-daemon.sh stop resourcemanager

原文地址:https://www.cnblogs.com/xiaotime/p/9534586.html

时间: 2024-10-03 07:31:13

hadoop集群之HDFS和YARN启动和停止命令的相关文章

为已存在的Hadoop集群配置HDFS Federation

一.实验目的 1. 现有Hadoop集群只有一个NameNode,现在要增加一个NameNode. 2. 两个NameNode构成HDFS Federation. 3. 不重启现有集群,不影响数据访问. 二.实验环境 4台CentOS release 6.4虚拟机,IP地址为 192.168.56.101 master 192.168.56.102 slave1 192.168.56.103 slave2 192.168.56.104 kettle 其中kettle是新增的一台"干净"

Hadoop集群(二) HDFS搭建

HDFS只是Hadoop最基本的一个服务,很多其他服务,都是基于HDFS展开的.所以部署一个HDFS集群,是很核心的一个动作,也是大数据平台的开始. 安装Hadoop集群,首先需要有Zookeeper才可以完成安装.如果没有Zookeeper,请先部署一套Zookeeper.另外,JDK以及物理主机的一些设置等.请参考: Hadoop集群(一) Zookeeper搭建 Hadoop集群(三) Hbase搭建 Hadoop集群(四) Hadoop升级 下面开始HDFS的安装 HDFS主机分配 1

Hadoop集群(第13期)_HBase 常用Shell命令

进入hbase shell console$HBASE_HOME/bin/hbase shell如果有kerberos认证,需要事先使用相应的keytab进行一下认证(使用kinit命令),认证成功之后再使用hbase shell进入可以使用whoami命令可查看当前用户 hbase(main)> whoami 表的管理1)查看有哪些表 hbase(main)> list 2)创建表 # 语法:create <table>, {NAME => <family>,

大数据系列(3)——Hadoop集群完全分布式坏境搭建

前言 上一篇我们讲解了Hadoop单节点的安装,并且已经通过VMware安装了一台CentOS 6.8的Linux系统,咱们本篇的目标就是要配置一个真正的完全分布式的Hadoop集群,闲言少叙,进入本篇的正题. 技术准备 VMware虚拟机.CentOS 6.8 64 bit 安装流程 我们先来回顾上一篇我们完成的单节点的Hadoop环境配置,已经配置了一个CentOS 6.8 并且完成了java运行环境的搭建,Hosts文件的配置.计算机名等诸多细节. 其实完成这一步之后我们就已经完成了Had

基于OGG的Oracle与Hadoop集群准实时同步介绍

Oracle里存储的结构化数据导出到Hadoop体系做离线计算是一种常见数据处置手段.近期有场景需要做Oracle到Hadoop体系的实时导入,这里以此案例做以介绍.Oracle作为商业化的数据库解决方案,自发性的获取数据库事务日志等比较困难,故选择官方提供的同步工具OGG(Oracle GoldenGate)来解决. 安装与基本配置 环境说明 软件配置 角色 数据存储服务及版本 OGG版本 IP 源服务器 OracleRelease11.2.0.1 Oracle GoldenGate 11.2

第124讲:Hadoop集群管理之fsimage和edits工作机制内幕详解学习笔记

客户端对hdfs进行写文件时会首先被记录在edits文件中. edits修改时元数据也会更新. 每次hdfs更新时edits先更新后客户端才会看到最新信息. fsimage:是namenode中关于元数据的镜像,一般称为检查点. 一般开始时对namenode的操作都放在edits中,为什么不放在fsimage中呢? 因为fsimage是namenode的完整的镜像,内容很大,如果每次都加载到内存的话生成树状拓扑结构,这是非常耗内存和CPU. 内容包含了namenode管理下的所有datanode

Hadoop集群性能优化一

挺喜欢这句话:"坚持,是基于 你对某件事的热爱,才能有动力坚持下去. 在学习的过程中,需要战胜自己的惰性和骄傲!"好了,下面说下如何提升 集群的性能: 在硬件方面,第一,商业硬件并不等同于低端硬件.低端机器常常使用 便宜的零部件,其故障率远高于更昂贵的机器.当用户管理几十台.上百台 甚至几千台机器时,便宜的零部件故障率更高,导致维护成本更高:第二, 不推荐使用大型数据库级别的机器,因为性价比太低了. 在相同硬件的情况下,一个配置好的的集群要比配置糟糕的集群在性能上 快数倍乃至数十倍.

Hadoop集群(第7期)_Eclipse开发环境设置

1.Hadoop开发环境简介 1.1 Hadoop集群简介 Java版本:jdk-6u31-linux-i586.bin Linux系统:CentOS6.0 Hadoop版本:hadoop-1.0.0.tar.gz 1.2 Windows开发简介 Java版本:jdk-6u31-windows-i586.exe Win系统:Windows 7 旗舰版 Eclipse软件:eclipse-jee-indigo-SR1-win32.zip | eclipse-jee-helios-SR2-win32

Hadoop集群_Eclipse开发环境设置

1.Hadoop开发环境简介 1.1 Hadoop集群简介 Java版本:jdk-6u31-linux-i586.bin Linux系统:CentOS6.0 Hadoop版本:hadoop-1.0.0.tar.gz 1.2 Windows开发简介 Java版本:jdk-6u31-windows-i586.exe Win系统:Windows 7 旗舰版 Eclipse软件:eclipse-jee-indigo-SR1-win32.zip | eclipse-jee-helios-SR2-win32