【kafka】一键启动kafka脚本

3.1 创建文件
cd bin 跳转到bin文件夹里
touch  start-kafka-cluster.sh --新建一键启动文件
touch  stop-kafka-cluster.sh --新建一键停止文件

3.2 编写start
start-kafka-cluster.sh内容:
#!/bin/bash
brokers="hadoop01 hadoop02 hadoop03"
KAFKA_HOME="/home/hadoop/kafka_2.12-2.3.0"
KAFKA_NAME="kafka_2.12-2.3.0"

echo "INFO : Begin to start kafka cluster ..."

for broker in $brokers
do
  echo "INFO : Starting ${KAFKA_NAME} on ${broker} ..."
  ssh ${broker} -C "source /etc/profile; sh ${KAFKA_HOME}/bin/kafka-server-start.sh -daemon ${KAFKA_HOME}/config/server.properties"
  if [[ $? -eq 0 ]]; then
      echo "INFO:[${broker}] Start successfully"
  fi
done
echo "INFO:Kafka cluster starts successfully !"

3.2 编写stop
stop-kafka-cluster.sh内容:
#!/bin/bash
brokers="hadoop01 hadoop02 hadoop03"
KAFKA_HOME="/home/hadoop/kafka_2.11-2.3.0"
KAFKA_NAME="kafka_2.11-2.3.0"
echo "INFO : Begin to stop kafka cluster ..."
for broker in $brokers
do
  echo "INFO : Shut down ${KAFKA_NAME} on ${broker} ..."
  ssh ${broker} "source /etc/profile;bash ${KAFKA_HOME}/bin/kafka-server-stop.sh"
  if [[ $? -ne 0 ]]; then
      echo "INFO : Shut down ${KAFKA_NAME} on ${broker} is down"
  fi
done
注1:如 cat -v start-kafka-cluster.sh 或 cat -v start-kafka-cluster.sh 遇见每行末尾有“^M”,则使用cat filename |tr -d ‘\r‘ > newfilename 命令
注2:修改kafka-server-stop.sh(因为安装后本身stop文件不能真正停止kafka进程):
#PIDS=$(ps ax | grep -i ‘kafka\.Kafka‘ | grep java | grep -v grep | awk ‘{print $1}‘)
PIDS=$(ps ax | grep -i ‘kafka\.Kafka‘ | awk ‘{print $1}‘ | xargs kill -SIGTERM)

具体执行过程如下:

[[email protected] ~]$ jps --查看kafka是否启动
8736 NodeManager
8593 ResourceManager
8083 DataNode
7942 NameNode
21336 Jps
8330 SecondaryNameNode
13595 Kafka
[[email protected] ~]$ cd bin
[[email protected] bin]$ sh stop-kafka-cluster.sh --启动stop脚本
INFO : Begin to stop kafka cluster ...
INFO : Shut down kafka_2.12-2.3.0 on hadoop01 ...
No kafka server to stop
INFO : Shut down kafka_2.12-2.3.0 on hadoop01 is down
INFO : Shut down kafka_2.12-2.3.0 on hadoop02 ...
No kafka server to stop
INFO : Shut down kafka_2.12-2.3.0 on hadoop02 is down
INFO : Shut down kafka_2.12-2.3.0 on hadoop03 ...
No kafka server to stop
INFO : Shut down kafka_2.12-2.3.0 on hadoop03 is down
INFO : kafka cluster shut down completed!
[[email protected] bin]$ jps --查看是否停止成功
8736 NodeManager
8593 ResourceManager
8083 DataNode
21428 Jps
7942 NameNode
8330 SecondaryNameNode
[[email protected] bin]$ sh start-kafka-cluster.sh --启动start脚本
INFO : Begin to start kafka cluster ...
INFO : Starting kafka_2.12-2.3.0 on hadoop01 ...
INFO:[hadoop01] Start successfully
INFO : Starting kafka_2.12-2.3.0 on hadoop02 ...
INFO:[hadoop02] Start successfully
INFO : Starting kafka_2.12-2.3.0 on hadoop03 ...
INFO:[hadoop03] Start successfully
INFO:Kafka cluster starts successfully !
[[email protected] bin]$ jps --查看节点1是否启动成功
8736 NodeManager
8593 ResourceManager
8083 DataNode
21923 Jps
7942 NameNode
21847 Kafka
8330 SecondaryNameNode
[[email protected] bin]$ ssh hadoop02 --查看节点2是否启动成功
Last login: Wed Oct  2 09:21:59 2019 from gateway
[[email protected] ~]$ jps
10564 NodeManager
10888 QuorumPeerMain
15563 Jps
10431 DataNode
15439 Kafka

原文地址:https://www.cnblogs.com/CQ-LQJ/p/11617134.html

时间: 2024-08-29 23:12:57

【kafka】一键启动kafka脚本的相关文章

Linux下安装并(单节点)配置启动Kafka

1. 从Kafka官网下载最新的Kafka,目前最新版本为0.9.0.1 2. 下载完毕后,上传到Linux服务器,并解压  tar -xzf kafka_2.11-0.9.0.1.tgz 3. 修改Zookeeper服务器配置,并启动 cd kafka_2.11-0.9.0.1 vi config/zookeeper.properties  #修改ZooKeeper的数据目录 dataDir=/opt/favccxx/db/zookeeper #配置host.name和advertised.h

windows安装kafka过程 【本地测试】外赋快速启动小脚本

winsows 上装kafka也蛮简单方便 下载:wget https://www.apache.org/dyn/closer.cgi?path=/kafka/2.1.0/kafka_2.12-2.1.0.tgz 解压到某文件下  ${kafka-path} 在  ${kafka-path}/bin 下 修改server.properties  中 : log.dirs=D:\\ant_JAR\\kafka_2.12-0.10.2.1\\kafka_log  #${kafka-path}下建立k

kafka的相关操作脚本

1.创建Topic./kafka-topics.sh --zookeeper cc13-141:2182 --topic mytopic --replication-factor 1 --partitions 1 --create 2.Topic列表./kafka-topics.sh --zookeeper cc13-138:2181 --list 3.创建生产者./kafka-console-producer.sh --broker-list cc13-141:9092 --topic myt

全网最详细的启动Kafka服务时出现kafka.common.InconsistentBrokerIdException: Configured brokerId 3 doesn't match stored brokerId 1 in meta.properties错误的解决办法(图文详解)

不多说,直接上干货! 问题详情 执行bin/kafka-server-start.sh config/server.properties 时, [[email protected] kafka_2.11-0.9.0.0]$ bin/kafka-server-start.sh config/server.properties [2018-06-17 16:05:38,983] INFO KafkaConfig values: request.timeout.ms = 30000 log.roll.

安装启动kafka

vim kafka/config/server.properties #确保唯一 broker.id=0 #允许删除主题 delete.topic.enable=true # 指定数据文件所在目录 log.dirs=/var/local/kafka/logs #zookeeper 的连接地址 zookeeper.connect=localhost:2181 启动kafka ./bin/kafka-server-start.sh config/server.properties #查看进程,带上全

首次启动Kafka报Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)

首次启动Kafka报错如下: 原因:内存不足,查看启动配置 调小一些: 首次启动Kafka报Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12) 原文地址:https://www.cnblogs.com/zjfjava/p/10325523.html

启动Kafka

启动zookeeper 使用命令查看zookeeper是否启动成功: 启动kafka Brokerr 使用命令查看kafka Broker是否启动更成功 在kafka中创建topic 'test' bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test1 列出所有的topic bin/kafka-topics.sh --list --zoo

启动kafka报错

执行启动kafka服务的命令时报错:nohup bin/kafka-server-start.sh config/server.properties 2>&1 & [2019-07-28 12:58:44,760] ERROR [ReplicaManager broker=1] Error while making broker the follower for partition Topic: __consumer_offsets; Partition: 41; Leader: N

大数据技术之_10_Kafka学习_Kafka概述+Kafka集群部署+Kafka工作流程分析+Kafka API实战+Kafka Producer拦截器+Kafka Streams

第1章 Kafka概述1.1 消息队列1.2 为什么需要消息队列1.3 什么是Kafka1.4 Kafka架构第2章 Kafka集群部署2.1 环境准备2.1.1 集群规划2.1.2 jar包下载2.2 Kafka集群部署2.3 Kafka命令行操作第3章 Kafka工作流程分析3.1 Kafka 生产过程分析3.1.1 写入方式3.1.2 分区(Partition)3.1.3 副本(Replication)3.1.4 写入流程3.2 Broker 保存消息3.2.1 存储方式3.2.2 存储策