/*************
*kafka 0.8.1.1的安装部署
*blog:www.r66r.net
*qq:26571864
**************/
相关部署视频地址：http://edu.51cto.com/course/course_id-2374.html
kafka的部署模式为3种模式
1)单broker模式
2)单机多broker模式 (伪集群)
3)多机多broker模式 (真正的集群模式)
第一种模式安装
1.在hadoopdn2机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面
2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=0 默认不用修改
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000 默认不用修改
zookeeper.connect=hadoopdn2:2181
3.启动kafka的broker
> bin/kafka-server-start.sh config/server.properties
正常启动如下：
[2014-11-18 10:36:32,196] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:os.version=2.6.32-220.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,196] INFO Client environment:user.dir=/opt/hadoop/kafka/kafka_2.9.2-0.8.1.1 (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,197] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 [email protected] (org.apache.zookeeper.ZooKeeper)
[2014-11-18 10:36:32,231] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,238] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,262] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x349c07dcd7a0002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-11-18 10:36:32,266] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-11-18 10:36:32,415] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2014-11-18 10:36:32,422] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-11-18 10:36:32,502] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-11-18 10:36:32,503] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2014-11-18 10:36:32,634] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-11-18 10:36:32,716] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-11-18 10:36:32,887] INFO Registered broker 0 at path /brokers/ids/0 with address JobTracker:9092. (kafka.utils.ZkUtils$)
[2014-11-18 10:36:32,941] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2014-11-18 10:36:33,034] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
4.创建topics
> bin/kafka-topics.sh --create --zookeeper hadoopdn2:2181 --replication-factor 1 --partitions 1 --topic test
查看队列列表
> bin/kafka-topics.sh --list --zookeeper hadoopdn2:2181
查看队列明细
> bin/kafka-topics.sh --describe --zookeeper hadoopdn2:2181 --topic test
Topic[队列]:test PartitionCount[分区数量]:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
第一行是所有Partition的总结。后面的行是每个partition一行。
查看帮助文档
> bin/kafka-topics.sh --help 查看与topics相关的指令
第二种模式部署：
1.为第二个broker创建server的配置文件
> cp server.properties server1.properties
2.修改server1.properties
broker.id=1
port=9093
log.dirs=/opt/hadoop/kafka/kafka-logs-server1
zookeeper.connect=hadoopdn2:2181
3.启动kafka的broker
> nohup bin/kafka-server-start.sh config/server1.properties &
4.通过zookeeper的客户端可以查看当前的broker
[zk: hadoopdn2:2181(CONNECTED) 7] ls /
[zookeeper, admin, consumers, config, controller, brokers, controller_epoch]
[zk: hadoopdn2:2181(CONNECTED) 8] ls /brokers
[topics, ids]
[zk: hadoopdn2:2181(CONNECTED) 9] ls /brokers/ids
[1, 0]
5.查看队列情况
$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
6.修改test队列的参数
$ bin/kafka-topics.sh --zookeeper hadoopdn2:2181 --partitions 3 --topic test --alter
$ bin/kafka-topics.sh --describe test --zookeeper hadoopdn2:2181
Topic:test PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0[在broker0上面] Isr: 0
Topic: test Partition: 1 Leader: 1 Replicas: 1[在broker1上面] Isr: 1
Topic: test Partition: 2 Leader: 0 Replicas: 0[在broker0上面] Isr: 0
第三种部署方式：
1.在hadoopdn3机器上面上传kafka文件，并解压到 /opt/hadoop/kafka下面
2.修改 /opt/hadoop/kafka/kafka_2.9.2-0.8.1.1/config 下面的server.properties 配置文件
broker.id=2 必须修改保证每个broker的ID唯一
修改
log.dirs=/opt/hadoop/kafka/kafka-logs
log.flush.interval.messages=10000 默认不用修改
log.flush.interval.ms=1000 默认不用修改
zookeeper.connect=hadoopdn2:2181
3.通过zookeeper的客户端查看
[zk: hadoopdn2:2181(CONNECTED) 10] ls /brokers/ids
[2, 1, 0]
broker的id为2的已经注册到zookeeper上面了
到此为止，kafka的部署模式已经完整。