Zookeeper原理 zk部署 应用 kafka

Top

NSD ARCHITECTURE DAY07

  1. 案例1：Zookeeper安装
  2. 步骤一：安装Zookeeper
  3. 案例2：Kafka集群实验
  4. 案例3：Hadoop高可用
  5. 案例4：高可用验证

1 案例1：Zookeeper安装

1.1 问题

本案例要求：

  • 搭建Zookeeper集群并查看各服务器的角色
  • 停止Leader并查看各服务器的角色

1.2 步骤

实现此案例需要按照如下步骤进行。

2 步骤一：安装Zookeeper

1）编辑/etc/hosts ,所有集群主机可以相互 ping 通（在nn01上面配置，同步到node1，node2，node3）

  1. [[email protected] hadoop]# vim /etc/hosts
  2. 192.168.1.21 nn01
  3. 192.168.1.22 node1
  4. 192.168.1.23 node2
  5. 192.168.1.24 node3
  6. 192.168.1.25 node4
  7. [[email protected] hadoop]# for i in {22..24} \
  8. do    \
  9. scp /etc/hosts 192.168.1.$i:/etc/    \
  10. done        //同步配置
  11. hosts 100% 253 639.2KB/s 00:00
  12. hosts 100% 253 497.7KB/s 00:00
  13. hosts 100% 253 662.2KB/s 00:00

2）安装 java-1.8.0-openjdk-devel,由于之前的hadoop上面已经安装过，这里不再安装，若是新机器要安装

3）zookeeper 解压拷贝到 /usr/local/zookeeper

  1. [[email protected] ~]# tar -xf zookeeper-3.4.10.tar.gz
  2. [[email protected] ~]# mv zookeeper-3.4.10 /usr/local/zookeeper

4）配置文件改名，并在最后添加配置

  1. [[email protected] ~]# cd /usr/local/zookeeper/conf/
  2. [[email protected] conf]# ls
  3. configuration.xsl log4j.properties zoo_sample.cfg
  4. [[email protected] conf]# mv zoo_sample.cfg zoo.cfg
  5. [[email protected] conf]# chown root.root zoo.cfg
  6. [[email protected] conf]# vim zoo.cfg
  7. server.1=node1:2888:3888
  8. server.2=node2:2888:3888
  9. server.3=node3:2888:3888
  10. server.4=nn01:2888:3888:observer

5）拷贝 /usr/local/zookeeper 到其他集群主机

  1. [[email protected] conf]# for i in {22..24}; do rsync -aSH --delete /usr/local/zookeeper/ 192.168.1.$i:/usr/local/zookeeper -e ‘ssh‘ & done
  2. [4] 4956
  3. [5] 4957
  4. [6] 4958

6）创建 mkdir /tmp/zookeeper，每一台都要

  1. [[email protected] conf]# mkdir /tmp/zookeeper
  2. [[email protected] conf]# ssh node1 mkdir /tmp/zookeeper
  3. [[email protected] conf]# ssh node2 mkdir /tmp/zookeeper
  4. [[email protected] conf]# ssh node3 mkdir /tmp/zookeeper

7）创建 myid 文件，id 必须与配置文件里主机名对应的 server.(id) 一致

  1. [[email protected] conf]# echo 4 >/tmp/zookeeper/myid
  2. [[email protected] conf]# ssh node1 ‘echo 1 >/tmp/zookeeper/myid‘
  3. [[email protected] conf]# ssh node2 ‘echo 2 >/tmp/zookeeper/myid‘
  4. [[email protected] conf]# ssh node3 ‘echo 3 >/tmp/zookeeper/myid‘

8）启动服务，单启动一台无法查看状态，需要启动全部集群以后才能查看状态，每一台上面都要手工启动（以nn01为例子）

  1. [[email protected] conf]# /usr/local/zookeeper/bin/zkServer.sh start
  2. ZooKeeper JMX enabled by default
  3. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  4. Starting zookeeper ... STARTED

注意：刚启动zookeeper查看状态的时候报错，启动的数量要保证半数以上，这时再去看就成功了

9）查看状态

  1. [[email protected] conf]# /usr/local/zookeeper/bin/zkServer.sh status
  2. ZooKeeper JMX enabled by default
  3. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  4. Mode: observe
  5. [[email protected] conf]# /usr/local/zookeeper/bin/zkServer.sh stop
  6. //关闭之后查看状态其他服务器的角色
  7. ZooKeeper JMX enabled by default
  8. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  9. Stopping zookeeper ... STOPPED
  10. [[email protected] conf]# yum -y install telnet
  11. [[email protected] conf]# telnet node3 2181
  12. Trying 192.168.1.24...
  13. Connected to node3.
  14. Escape character is ‘^]‘.
  15. ruok        //发送
  16. imokConnection closed by foreign host.        //imok回应的结果

10）利用 api 查看状态（nn01上面操作）

  1. [[email protected] conf]# /usr/local/zookeeper/bin/zkServer.sh start
  2. [[email protected] conf]# vim api.sh
  3. #!/bin/bash
  4. function getstatus(){
  5. exec 9<>/dev/tcp/$1/2181 2>/dev/null
  6. echo stat >&9
  7. MODE=$(cat <&9 |grep -Po "(?<=Mode:).*")
  8. exec 9<&-
  9. echo ${MODE:-NULL}
  10. }
  11. for i in node{1..3} nn01;do
  12. echo -ne "${i}\t"
  13. getstatus ${i}
  14. done
  15. [[email protected] conf]# chmod 755 api.sh
  16. [[email protected] conf]# ./api.sh
  17. node1    follower
  18. node2    leader
  19. node3    follower
  20. nn01    observer

3 案例2：Kafka集群实验

3.1 问题

本案例要求：

  • 利用Zookeeper搭建一个Kafka集群
  • 创建一个topic
  • 模拟生产者发布消息
  • 模拟消费者接收消息

3.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：搭建Kafka集群

1）解压 kafka 压缩包

Kafka在node1，node2，node3上面操作即可

  1. [[email protected] ~]# tar -xf kafka_2.10-0.10.2.1.tgz

2）把 kafka 拷贝到 /usr/local/kafka 下面

  1. [[email protected] ~]# mv kafka_2.10-0.10.2.1 /usr/local/kafka

3）修改配置文件 /usr/local/kafka/config/server.properties

  1. [[email protected] ~]# cd /usr/local/kafka/config
  2. [[email protected] config]# vim server.properties
  3. broker.id=22
  4. zookeeper.connect=node1:2181,node2:2181,node3:2181

4）拷贝 kafka 到其他主机，并修改 broker.id ,不能重复

  1. [[email protected] config]# for i in 23 24; do rsync -aSH --delete /usr/local/kafka 192.168.1.$i:/usr/local/; done
  2. [1] 27072
  3. [2] 27073
  4. [[email protected] ~]# vim /usr/local/kafka/config/server.properties
  5. //node2主机修改
  6. broker.id=23
  7. [[email protected] ~]# vim /usr/local/kafka/config/server.properties
  8. //node3主机修改
  9. broker.id=24

5）启动 kafka 集群（node1，node2，node3启动）

  1. [[email protected] local]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
  2. [[email protected] local]# jps        //出现kafka
  3. 26483 DataNode
  4. 27859 Jps
  5. 27833 Kafka
  6. 26895 QuorumPeerMain

6）验证配置，创建一个 topic

  1. [[email protected] local]# /usr/local/kafka/bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --zookeeper node3:2181 --topic aa
  2. Created topic "aa".

7) 模拟生产者，发布消息

  1. [[email protected] ~]# /usr/local/kafka/bin/kafka-console-producer.sh \
  2. --broker-list node2:9092 --topic aa        //写一个数据
  3. ccc
  4. ddd

9）模拟消费者，接收消息

  1. [[email protected] ~]# /usr/local/kafka/bin/kafka-console-consumer.sh \
  2. --bootstrap-server node1:9092 --topic aa        //这边会直接同步
  3. ccc
  4. ddd

注意：kafka比较吃内存，做完这个kafka的实验可以把它停了

4 案例3：Hadoop高可用

4.1 问题

本案例要求：

  • 配置Hadoop的高可用
  • 修改配置文件

4.2 方案

配置Hadoop的高可用，解决NameNode单点故障问题，使用之前搭建好的hadoop集群，新添加一台nn02，ip为192.168.1.25，之前有一台node4主机，可以用这台主机，具体要求如图-1所示：

图-1

4.3 步骤

实现此案例需要按照如下步骤进行。

步骤一：hadoop的高可用

1）停止所有服务（由于 kafka的实验做完之后就已经停止，这里不在重复）

  1. [[email protected] ~]# cd /usr/local/hadoop/
  2. [[email protected] hadoop]# ./sbin/stop-all.sh //停止所有服务

2）启动zookeeper（需要一台一台的启动）这里以nn01为例子

  1. [[email protected] hadoop]# /usr/local/zookeeper/bin/zkServer.sh start
  2. [[email protected] hadoop]# sh /usr/local/zookeeper/conf/api.sh //利用之前写好的脚本查看
  3. node1    follower
  4. node2    leader
  5. node3    follower
  6. nn01    observer

3）新加一台机器nn02，这里之前有一台node4，可以用这个作为nn02

  1. [[email protected] ~]# echo nn02 > /etc/hostname
  2. [[email protected] ~]# hostname nn02

4）修改vim /etc/hosts

  1. [[email protected] hadoop]# vim /etc/hosts
  2. 192.168.1.21 nn01
  3. 192.168.1.25 nn02
  4. 192.168.1.22 node1
  5. 192.168.1.23 node2
  6. 192.168.1.24 node3

5）同步到nn02，node1，node2，node3

  1. [[email protected] hadoop]# for i in {22..25}; do rsync -aSH --delete /etc/hosts 192.168.1.$i:/etc/hosts -e ‘ssh‘ & done
  2. [1] 14355
  3. [2] 14356
  4. [3] 14357
  5. [4] 14358

6）配置SSH信任关系

注意：nn01和nn02互相连接不需要密码，nn02连接自己和node1，node2，node3同样不需要密码

  1. [[email protected] ~]# vim /etc/ssh/ssh_config
  2. Host *
  3. GSSAPIAuthentication yes
  4. StrictHostKeyChecking no
  5. [[email protected] hadoop]# cd /root/.ssh/
  6. [[email protected] .ssh]# scp id_rsa id_rsa.pub nn02:/root/.ssh/
  7. //把nn01的公钥私钥考给nn02

7）所有的主机删除/var/hadoop/*

  1. [[email protected] .ssh]# rm -rf /var/hadoop/*
  2. [[email protected] .ssh]# ssh nn02 rm -rf /var/hadoop/*
  3. [[email protected] .ssh]# ssh node1 rm -rf /var/hadoop/*
  4. [[email protected] .ssh]# ssh node2 rm -rf /var/hadoop/*
  5. [[email protected] .ssh]# ssh node3 rm -rf /var/hadoop/*

8）配置 core-site

  1. [[email protected] .ssh]# vim /usr/local/hadoop/etc/hadoop/core-site.xml
  2. <configuration>
  3. <property>
  4. <name>fs.defaultFS</name>
  5. <value>hdfs://nsdcluster</value>
  6. //nsdcluster是随便起的名。相当于一个组，访问的时候访问这个组
  7. </property>
  8. <property>
  9. <name>hadoop.tmp.dir</name>
  10. <value>/var/hadoop</value>
  11. </property>
  12. <property>
  13. <name>ha.zookeeper.quorum</name>
  14. <value>node1:2181,node2:2181,node3:2181</value>    //zookeepe的地址
  15. </property>
  16. <property>
  17. <name>hadoop.proxyuser.nfs.groups</name>
  18. <value>*</value>
  19. </property>
  20. <property>
  21. <name>hadoop.proxyuser.nfs.hosts</name>
  22. <value>*</value>
  23. </property>
  24. </configuration>

9）配置 hdfs-site

  1. [[email protected] ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  2. <configuration>
  3. <property>
  4. <name>dfs.replication</name>
  5. <value>2</value>
  6. </property>
  7. <property>
  8. <name>dfs.nameservices</name>
  9. <value>nsdcluster</value>
  10. </property>
  11. <property>
  12. <name>dfs.ha.namenodes.nsdcluster</name>
  13. //nn1,nn2名称固定，是内置的变量，nsdcluster里面有nn1，nn2
  14. <value>nn1,nn2</value>
  15. </property>
  16. <property>
  17. <name>dfs.namenode.rpc-address.nsdcluster.nn1</name>
  18. //声明nn1 8020为通讯端口，是nn01的rpc通讯端口
  19. <value>nn01:8020</value>
  20. </property>
  21. <property>
  22. <name>dfs.namenode.rpc-address.nsdcluster.nn2</name>
  23. //声明nn2是谁，nn02的rpc通讯端口
  24. <value>nn02:8020</value>
  25. </property>
  26. <property>
  27. <name>dfs.namenode.http-address.nsdcluster.nn1</name>
  28. //nn01的http通讯端口
  29. <value>nn01:50070</value>
  30. </property>
  31. <property>
  32. <name>dfs.namenode.http-address.nsdcluster.nn2</name>
  33. //nn01和nn02的http通讯端口
  34. <value>nn02:50070</value>
  35. </property>
  36. <property>
  37. <name>dfs.namenode.shared.edits.dir</name>
  38. //指定namenode元数据存储在journalnode中的路径
  39. <value>qjournal://node1:8485;node2:8485;node3:8485/nsdcluster</value>
  40. </property>
  41. <property>
  42. <name>dfs.journalnode.edits.dir</name>
  43. //指定journalnode日志文件存储的路径
  44. <value>/var/hadoop/journal</value>
  45. </property>
  46. <property>
  47. <name>dfs.client.failover.proxy.provider.nsdcluster</name>
  48. //指定HDFS客户端连接active namenode的java类
  49. <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  50. </property>
  51. <property>
  52. <name>dfs.ha.fencing.methods</name>                    //配置隔离机制为ssh
  53. <value>sshfence</value>
  54. </property>
  55. <property>
  56. <name>dfs.ha.fencing.ssh.private-key-files</name>    //指定密钥的位置
  57. <value>/root/.ssh/id_rsa</value>
  58. </property>
  59. <property>
  60. <name>dfs.ha.automatic-failover.enabled</name>        //开启自动故障转移
  61. <value>true</value>
  62. </property>
  63. </configuration>

10）配置yarn-site

  1. [[email protected] ~]# vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
  2. <configuration>
  3. <!-- Site specific YARN configuration properties -->
  4. <property>
  5. <name>yarn.nodemanager.aux-services</name>
  6. <value>mapreduce_shuffle</value>
  7. </property>
  8. <property>
  9. <name>yarn.resourcemanager.ha.enabled</name>
  10. <value>true</value>
  11. </property>
  12. <property>
  13. <name>yarn.resourcemanager.ha.rm-ids</name>        //rm1,rm2代表nn01和nn02
  14. <value>rm1,rm2</value>
  15. </property>
  16. <property>
  17. <name>yarn.resourcemanager.recovery.enabled</name>
  18. <value>true</value>
  19. </property>
  20. <property>
  21. <name>yarn.resourcemanager.store.class</name>
  22. <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  23. </property>
  24. <property>
  25. <name>yarn.resourcemanager.zk-address</name>
  26. <value>node1:2181,node2:2181,node3:2181</value>
  27. </property>
  28. <property>
  29. <name>yarn.resourcemanager.cluster-id</name>
  30. <value>yarn-ha</value>
  31. </property>
  32. <property>
  33. <name>yarn.resourcemanager.hostname.rm1</name>
  34. <value>nn01</value>
  35. </property>
  36. <property>
  37. <name>yarn.resourcemanager.hostname.rm2</name>
  38. <value>nn02</value>
  39. </property>
  40. </configuration>

11）同步到nn02，node1，node2，node3

  1. [[email protected] ~]# for i in {22..25}; do rsync -aSH --delete /usr/local/hadoop/ 192.168.1.$i:/usr/local/hadoop -e ‘ssh‘ & done
  2. [1] 25411
  3. [2] 25412
  4. [3] 25413
  5. [4] 25414

12）删除所有机器上面的/user/local/hadoop/logs，方便排错

  1. [[email protected] ~]# for i in {21..25}; do ssh 192.168.1.$i rm -rf /usr/local/hadoop/logs ; done

13）同步配置

  1. [[email protected] ~]# for i in {22..25}; do rsync -aSH --delete /usr/local/hadoop 192.168.1.$i:/usr/local/hadoop -e ‘ssh‘ & done
  2. [1] 28235
  3. [2] 28236
  4. [3] 28237
  5. [4] 28238

5 案例4：高可用验证

5.1 问题

本案例要求：

  • 初始化集群
  • 验证集群

5.2 步骤

实现此案例需要按照如下步骤进行。

步骤一：验证hadoop的高可用

1）初始化ZK集群

  1. [[email protected] ~]# /usr/local/hadoop/bin/hdfs zkfc -formatZK
  2. ...
  3. 18/09/11 15:43:35 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/nsdcluster in ZK //出现Successfully即为成功
  4. ...

2）在node1，node2，node3上面启动journalnode服务（以node1为例子）

  1. [[email protected] ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
  2. starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node1.out
  3. [[email protected] ~]# jps
  4. 29262 JournalNode
  5. 26895 QuorumPeerMain
  6. 29311 Jps

3）格式化，先在node1，node2，node3上面启动journalnode才能格式化

  1. [[email protected] ~]# /usr/local/hadoop//bin/hdfs namenode -format
  2. //出现Successfully即为成功
  3. [[email protected] hadoop]# ls /var/hadoop/
  4. dfs

4）nn02数据同步到本地 /var/hadoop/dfs

  1. [[email protected] ~]# cd /var/hadoop/
  2. [[email protected] hadoop]# ls
  3. [[email protected] hadoop]# rsync -aSH nn01:/var/hadoop/ /var/hadoop/
  4. [[email protected] hadoop]# ls
  5. dfs

5）初始化 JNS

  1. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs namenode -initializeSharedEdits
  2. 18/09/11 16:26:15 INFO client.QuorumJournalManager: Successfully started new epoch 1        //出现Successfully，成功开启一个节点

6）停止 journalnode 服务（node1，node2，node3）

  1. [[email protected] hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop journalnode
  2. stopping journalnode
  3. [[email protected] hadoop]# jps
  4. 29346 Jps
  5. 26895 QuorumPeerMain

步骤二：启动集群

1）nn01上面操作

  1. [[email protected] hadoop]# /usr/local/hadoop/sbin/start-all.sh //启动所有集群
  2. This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
  3. Starting namenodes on [nn01 nn02]
  4. nn01: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-nn01.out
  5. nn02: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-nn02.out
  6. node2: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node2.out
  7. node3: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node3.out
  8. node1: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-node1.out
  9. Starting journal nodes [node1 node2 node3]
  10. node1: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node1.out
  11. node3: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node3.out
  12. node2: starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node2.out
  13. Starting ZK Failover Controllers on NN hosts [nn01 nn02]
  14. nn01: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-nn01.out
  15. nn02: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-nn02.out
  16. starting yarn daemons
  17. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-nn01.out
  18. node2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node2.out
  19. node1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node1.out
  20. node3: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-node3.out

2）nn02上面操作

  1. [[email protected] hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
  2. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-nn02.out

3）查看集群状态

  1. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  2. active
  3. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
  4. standby
  5. [[email protected] hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  6. active
  7. [[email protected] hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
  8. standby

4）查看节点是否加入

  1. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs dfsadmin -report
  2. ...
  3. Live datanodes (3): //会有三个节点
  4. ...
  5. [[email protected] hadoop]# /usr/local/hadoop/bin/yarn node -list
  6. Total Nodes:3
  7. Node-Id     Node-State    Node-Http-Address    Number-of-Running-Containers
  8. node2:43307     RUNNING     node2:8042     0
  9. node1:34606     RUNNING     node1:8042     0
  10. node3:36749     RUNNING     node3:8042     0

步骤三：访问集群

1）查看并创建

  1. [[email protected] hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /
  2. [[email protected] hadoop]# /usr/local/hadoop/bin/hadoop fs -mkdir /aa //创建aa
  3. [[email protected] hadoop]# /usr/local/hadoop/bin/hadoop fs -ls /        //再次查看
  4. Found 1 items
  5. drwxr-xr-x - root supergroup 0 2018-09-11 16:54 /aa
  6. [[email protected] hadoop]# /usr/local/hadoop/bin/hadoop fs -put *.txt /aa
  7. [[email protected] hadoop]# /usr/local/hadoop/bin/hadoop fs -ls hdfs://nsdcluster/aa
  8. //也可以这样查看
  9. Found 3 items
  10. -rw-r--r-- 2 root supergroup 86424 2018-09-11 17:00 hdfs://nsdcluster/aa/LICENSE.txt
  11. -rw-r--r-- 2 root supergroup 14978 2018-09-11 17:00 hdfs://nsdcluster/aa/NOTICE.txt
  12. -rw-r--r-- 2 root supergroup 1366 2018-09-11 17:00 hdfs://nsdcluster/aa/README.txt

2）验证高可用，关闭 active namenode

  1. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  2. active
  3. [[email protected] hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop namenode
  4. stopping namenode
  5. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  6. //再次查看会报错
  7. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
  8. //nn02由之前的standby变为active
  9. active
  10. [[email protected] hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  11. active
  12. [[email protected] hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh stop resourcemanager
  13. //停止resourcemanager
  14. [[email protected] hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
  15. active

3） 恢复节点

  1. [[email protected] hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh start namenode
  2. //启动namenode
  3. [[email protected] hadoop]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
  4. //启动resourcemanager
  5. [[email protected] hadoop]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
  6. //查看
  7. [[email protected] hadoop]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
  8. //查看

原文地址：https://www.cnblogs.com/tiki/p/10785624.html

时间： 2024-10-18 07:16:10

Zookeeper原理 zk部署 应用 kafka的相关文章

kafka集群和zookeeper集群的部署，kafka的java代码示例

来自:http://doc.okbase.net/QING____/archive/19447.html 也可参考: http://blog.csdn.net/21aspnet/article/details/19325373 http://blog.csdn.net/unix21/article/details/18990123 kafka作为分布式日志收集或系统监控服务,我们有必要在合适的场合使用它.kafka的部署包括zookeeper环境/kafka环境,同时还需要进行一些配置操作.接下

ZooKeeper原理分析

ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,它包含一个简单的原语集,分布式应用程序可以基于它实现同步服务,配置维护和命名服务等.Zookeeper是hadoop的一个子项目,其发展历程无需赘述.在分布式应用中,由于工程师不能很好地使用锁机制,以及基于消息的协调机制不适合在某些应用中使用,因此需要有一种可靠的.可扩展的.分布式的.可配置的协调机制来统一系统的状态.Zookeeper的目的就在于此.本文简单分析zookeeper的工作原理,对于如何使用zookeeper不是本

zookeeper安装及部署

阅读目录 一. 单机安装.配置 二. 集群安装.配置 安装及部署 回到顶部 一. 单机安装.配置 1.下载zookeeper二进制安装包 下载 curl -L -O http://apache.fayea.com/zookeeper/stable/zookeeper-3.4.6.tar.gz 解压 tar zxvf zookeeper-3.4.6.tar.gz 2.设置环境变量 ZOOKEEPER_HOME关键字为添加的配置项 # /etc/profile # System wide envir

8.8.ZooKeeper 原理和选举机制

1.ZooKeeper原理 Zookeeper虽然在配置文件中并没有指定master和slave但是,zookeeper工作时,是有一个节点为leader,其他则为follower,Leader是通 过内部的选举机制临时产生的 2.ZooKeeper选举机制 2.1.概念 2.2. zk的选举机制(全新集群paxos):新搭建,无任何数据(通过myid) 2.3. 非全新集群的选举机制(数据恢复) 原文地址:https://www.cnblogs.com/yaboya/p/9149077.htm

Zookeeper集群部署和使用

Zookeeper 由 Apache Hadoop 的 Zookeeper 子项目发展而来,Google Chubby的一个开源实现.它是一个分布式应用程序协调服务,提供的功能包括:配置管理,名字服务,提供分布式同步.队列管理.集群管理等. 从设计模式角度来看,是一个基于观察者模式设计的分布式服务管理框架,它负责存储和管理大家都关心的数据,然后接受观察者的注册,一旦这些数据的状态发生变化,Zookeeper 就将负责通知已经在 Zookeeper 上注册的那些观察者做出相应的反应,从而实现集群中

zookeeper集群部署总结篇

前前后后各种问题不断终于实现集群部署,特做如下记录,因为zookeeper 和 kafka 都依赖于java 环境,因此需要装jdk. 通过ftp 上传   apache-zookeeper-3.5.6-bin.tar.gz   jdk-8u241-linux-x64.tar.gz  kafka_2.13-2.4.0.tgz  jdk 环境配置 1.mv jdk-8u162-linux-x64.tar.gz /usr/local 2.tar -zxvf jdk-8u162-linux-x64.t

离线部署ELK+kafka日志管理系统

1.简介 对于日志来说,最常见的需求就是收集.查询.显示,正对应logstash.elasticsearch.kibana的功能. ELK日志系统在系统中,主要可解决的问题: 基于日志的数据挖掘 问题排查,上线检查 根据关键字查询日志详情 异常数据自动触发消息通知 服务器监控,应用监控,Bug管理 统计分析,比如接口的调用次数.执行时间.成功率等 性能分析,用户行为分析,安全漏洞分析,时间管理 Logstash: Logstash是一个用来搜集.分析.过滤日志的工具.它支持几乎任何类型的日志,包

浅谈 zookeeper 原理,安装和配置

当前云计算流行, 单一机器额的处理能力已经不能满足我们的需求,不得不采用大量的服务集群.服务集群对外提供服务的过程中,有很多的配置需要随时更新,服务间需要协调工作,那么这些信息如何推送到各个节点?并且保证信息的一致性和可靠性?我们知道分布式协调服务很难正确无误地实现, 因为他们很容易在竞争条件和死锁上犯错误.那么zookeeper将是一个不错的选择 Zookeeper是什么? 引用官方的说法:“Zookeeper是一个高性能,分布式的,开源分布式应用协调服务.它提供了简单原始的功能,分布式应用可

(转)heartbeat原理及部署

原文:http://yjy724.blog.51cto.com/10897133/1840794---------------------------------------------------heartbeat原理及部署 1. Heartbeat介绍 1.1 Heartbeat作用 通过它可以将资源(IP及程序服务等资源)从一台故障计算机快速转移到另一台运转正常的机器继续提供服务,在实际生产应用场景中,heartbeat的功能和另一个高可用开源软件keepalived有很多相同之处 1.2