kafka 扩展partition和replication-factor

问题:

1. kafka的topic 是程序自己建立,默认只建立8个partitions,1个replication-factor

目的: 扩展partitions 到9个, replicatoion-factor  3个

原因:因为kafka是三台,如果replicatoion-factor设置1的话,kafka坏掉一个会有影响

(1). 扩展partitions 到9个

/kafka-topics.sh --zookeeper kafka-zookeeper01:2181,kafka-zookeeper02:2181,kafka-zookeeper03:2181/kafka --alter --topic Message --partitions 9

查看结果:

/kafka-topics.sh --describe  --zookeeper kafka-zookeeper01:2181,kafka-zookeeper02:2181,kafka-zookeeper03:2181/kafka  --topic Message

Topic:Message  PartitionCount:9 ReplicationFactor:1 Configs:
Topic: Message  Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic:Message  Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: Message  Partition: 2 Leader: 3 Replicas: 3 Isr: 3
Topic: Message  Partition: 3 Leader: 1 Replicas: 1 Isr: 1
Topic: Message Partition: 4 Leader: 2 Replicas: 2 Isr: 2
Topic: Message  Partition: 5 Leader: 3 Replicas: 3 Isr: 3
Topic: Message Partition: 6 Leader: 1 Replicas: 1 Isr: 1
Topic: Message  Partition: 7 Leader: 2 Replicas: 2 Isr: 2

Topic: Message  Partition: 8 Leader: 3 Replicas: 3 Isr: 3

(2)扩展ReplicationFactor 为3

vim increace-factor.json

{"version":1,
"partitions":[
{"topic":"Message","partition":0,"replicas":[1,2,3]},
{"topic":"Message","partition":1,"replicas":[1,2,3]},
{"topic":"Message","partition":2,"replicas":[1,2,3]},
{"topic":"Message","partition":3,"replicas":[1,2,3]},
{"topic":"Message","partition":4,"replicas":[1,2,3]},
{"topic":"Message","partition":5,"replicas":[1,2,3]},
{"topic":"Message","partition":6,"replicas":[1,2,3]},
{"topic":"Message","partition":7,"replicas":[1,2,3]}

{"topic":"Message","partition":8,"replicas":[1,2,3]}
]}

执行如下命令:

/kafka-reassign-partitions.sh --zookeeper kafka-zookeeper01:2181,kafka-zookeeper02:2181,kafka-zookeeper03:2181/kafka --reassignment-json-file increace-factor.json --execute

[[email protected]]#/kafka-topics.sh --describe --zookeeper kafka-zookeeper01:2181,kafka-zookeeper02:2181,kafka-zookeeper03:2181/kafka  --topic Message
Topic:Message PartitionCount:9 ReplicationFactor:3 Configs:
Topic: Message Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: Message Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 2,1,3
Topic: Message Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 3,1,2
Topic: Message Partition: 3 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: Message Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 2,1,3
Topic: Message  Partition: 5 Leader: 1 Replicas: 1,2,3 Isr: 3,1,2
Topic: Message Partition: 6 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: cMessage Partition: 7 Leader: 1 Replicas: 1,2,3 Isr: 2,1,3

Topic: Message  Partition: 8 Leader: 1 Replicas: 1,2,3 Isr: 2,1,3

原文地址:https://www.cnblogs.com/wangyh702/p/11477347.html

时间: 2024-10-11 10:24:37

kafka 扩展partition和replication-factor的相关文章

Error while executing topic command : Replication factor: 2 larger than available brokers: 0.

[[email protected] /mnt/software/maxwell-1.19.4]#kafka-topics.sh --zookeeper hdp1,hdp2,hdp3:2181 --create --topic maxwell --partitions 6 --replication-factor 2Error while executing topic command : Replication factor: 2 larger than available brokers:

【kafka】kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0

https://blog.csdn.net/bigtree_3721/article/details/78442912 I am trying to create topics in Kafka by following the guide on Apache Kafka website through command line. While running the command:bin/kafka-topics.sh --create --zookeeper localhost:2181 -

kafka之partition分区及副本replica升级

修改kafka的partition分区 bin/kafka-topics.sh --zookeeper datacollect-2:2181 --alter --partitions 3 --topic client-agent-1 修改kafka副本数 官网解释如下: Increasing replication factor Increasing the replication factor of an existing partition is easy. Just specify the

Kafka Topic Partition Replica Assignment实现原理及资源隔离方案

本文共分为三个部分: Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案 1. Kafka Topic创建方式 Kafka Topic创建方式有以下两种表现形式: (1)创建Topic时直接指定Topic Partition Replica与Kafka Broker之间的存储映射关系 /usr/lib/kafka_2.10-0.8.2.1/bin/kafka-topics.sh --zookeeper ZooKeep

linux 下安装 php kafka 扩展

我们使用官方推荐 php kafka 扩展 phpkafka,由于该扩展是基于 librdkafka 开发,所以我们首先需要安装 librdkafka 下载地址:http://kafka.apache.org/downloads PHP 扩展:https://github.com/arnaud-lb/php-rdkafka 1.解压 librdkafka,执行 linux 下安装三部曲:./configure --> make --> make install ,此时在 /usr/local/

zookeeper和PHP zookeeper和kafka 扩展安装

http://blog.csdn.net/fenglailea/article/details/52458737#t3 目录(?)[-] 安装zookeeper 1直接安装zookeeper无须编译 2源码编译安装zookeeper 安装php zookeeper扩展 注意最新版kafka请使用73和4PASS 安装librdkafka 安装php-kafka扩展 配置扩展 查看是否安装完成 PHP-kafka扩展3和4安装完成后无法使用时 风来了.fox 安装zookeeper 推荐1.1 直

kafka sink partition

看如下代码段,kafka sink的key完全取决于上游发来的event中的header.所以,如果前面的source是像exec这样的source,由于其中的header为null,所以到了这就发到kafka中就是没key的.没key的情况下,kafka就做不了到不同broker的partition操作.

kafka中partition和消费者对应关系

1个partition只能被同组的一个consumer消费,同组的consumer则起到均衡效果 消费者多于partition topic: test 只有一个partition创建一个topic——test, bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test 在g2组中启动两个consumer, 1. bin/kafka-conso

Ubuntu16.04安装php kafka扩展

一.安装librdkafka git clone https://github.com/edenhill/librdkafka.gitcd librdkafka/./configuremakesudo make install //甚至不用这一步cd /usr/local/lib/sudo ln -sf librdkafka++.so.1 librdkafka++.so 二.安装扩展git clone https://github.com/arnaud-lb/php-rdkafka.gitcd