简洁的kafka单节点和分布式安装文档 / 憋错料

单节点安装:

Ip:single:192.168.1.186

1.下载:http://kafka.apache.org/downloads.html 下载源码或者对应scala版本的bin包

2.解压

[[email protected] cloud]# tar -zxvf kafka_2.10-0.8.2-beta.tgz -C /cloud/

3.进入解压目录

[[email protected] kafka_2.10-0.8.2-beta]# cd config/

[[email protected] config]# ll

total 32

-rw-rw-r--. 1 root root 1199 Oct 21 10:30 consumer.properties

-rw-rw-r--. 1 root root 3846 Oct 21 10:30 log4j.properties

-rw-rw-r--. 1 root root 2228 Oct 21 10:30 producer.properties

-rw-rw-r--. 1 root root 5558 Oct 21 10:30 server.properties

-rw-rw-r--. 1 root root 3325 Oct 21 10:30 test-log4j.properties

-rw-rw-r--. 1 root root 993 Oct 21 10:30 tools-log4j.properties

-rw-rw-r--. 1 root root 1023 Oct 21 10:30 zookeeper.properties

4.更改服务器配置

修改服务器的config/server.properties

broker.id：唯一，填数字，本文中分别为132/133/134

host.name：唯一，填服务器IP

zookeeper.connect=192.168.1.186:2181 配置zk的地址,此处是个单节点的zk

5.启动服务器

先启动zookeeper服务: bin/zkServer.sh start （本文中zookeeper为独立安装，具体过程在此不细述）

[[email protected] bin]# jps

4626 QuorumPeerMain

2267 RunJar

4895 Jps

再在每台机器上执行： bin/kafka-server-start.sh config/server.properties

启动报错:

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-server-start.sh config/server.properties

Unrecognized VM option ‘UseCompressedOops‘

Error: Could not create the Java Virtual Machine.

Error: A fatal exception has occurred. Program will exit.

解决方法:

找到bin/kafka-run-class.sh 文件,使用vim打开,我的这个版本是在115行

113 # JVM performance options

114 if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then

115 KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRe mark -XX:+DisableExplicitGC -Djava.awt.headless=true"

116 fi

去掉-XX:+UseCompressedOops这个设置

启动

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-server-start.sh config/server.properties

[2015-01-24 22:21:48,867] INFO Verifying properties (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,064] INFO Property broker.id is overridden to 1 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,064] INFO Property host.name is overridden to single (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,064] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,065] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,065] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,065] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,065] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,065] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,066] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,066] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,066] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,066] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,066] INFO Property socket.receive.buffer.bytes is overridden to 65536 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,067] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,067] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,067] INFO Property zookeeper.connect is overridden to single:2181 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,067] INFO Property zookeeper.connection.timeout.ms is overridden to 2000 (kafka.utils.VerifiableProperties)

[2015-01-24 22:21:49,154] INFO [Kafka Server 1], starting (kafka.server.KafkaServer)

[2015-01-24 22:21:49,157] INFO [Kafka Server 1], Connecting to zookeeper on single:2181 (kafka.server.KafkaServer)

[2015-01-24 22:21:49,191] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)

[2015-01-24 22:21:49,202] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:host.name=single (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.version=1.7.0_25 (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.home=/usr/local/jdk1.6.0_45/jre (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.class.path=:/cloud/kafka_2.10-0.8.2-beta/bin/../core/build/dependant-libs-2.10.1*/*.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../examples/build/libs//kafka-examples*.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../clients/build/libs/kafka-clients*.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/jopt-simple-3.2.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/kafka_2.10-0.8.2-beta.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/kafka_2.10-0.8.2-beta-javadoc.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/kafka_2.10-0.8.2-beta-scaladoc.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/kafka_2.10-0.8.2-beta-sources.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/kafka_2.10-0.8.2-beta-test.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/kafka-clients-0.8.2-beta.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/log4j-1.2.16.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/lz4-1.2.0.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/metrics-core-2.2.0.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/scala-library-2.10.1.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/slf4j-api-1.7.6.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/slf4j-log4j12-1.6.1.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/snappy-java-1.1.1.3.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/zkclient-0.3.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../libs/zookeeper-3.4.6.jar:/cloud/kafka_2.10-0.8.2-beta/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.library.path=/usr/java/packages/lib/i386:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:os.arch=i386 (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:os.version=2.6.32-358.el6.i686 (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,202] INFO Client environment:user.dir=/cloud/kafka_2.10-0.8.2-beta (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,210] INFO Initiating client connection, connectString=single:2181 sessionTimeout=6000 [email protected] (org.apache.zookeeper.ZooKeeper)

[2015-01-24 22:21:49,319] INFO Opening socket connection to server single/192.168.1.186:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2015-01-24 22:21:49,338] INFO Socket connection established to single/192.168.1.186:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2015-01-24 22:21:49,373] INFO Session establishment complete on server single/192.168.1.186:2181, sessionid = 0x14b1f20915b0002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)

[2015-01-24 22:21:49,387] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)

[2015-01-24 22:21:49,500] INFO Loading logs. (kafka.log.LogManager)

[2015-01-24 22:21:49,516] INFO Logs loading complete. (kafka.log.LogManager)

[2015-01-24 22:21:49,517] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)

[2015-01-24 22:21:49,522] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)

[2015-01-24 22:21:49,593] INFO Awaiting socket connections on single:9092. (kafka.network.Acceptor)

[2015-01-24 22:21:49,598] INFO [Socket Server on Broker 1], Started (kafka.network.SocketServer)

[2015-01-24 22:21:49,748] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2015-01-24 22:21:49,859] INFO 1 successfully elected as leader (kafka.server.ZookeeperLeaderElector)

[2015-01-24 22:21:50,045] INFO Registered broker 1 at path /brokers/ids/1 with address single:9092. (kafka.utils.ZkUtils$)

[2015-01-24 22:21:50,102] INFO [Kafka Server 1], started (kafka.server.KafkaServer)

[2015-01-24 22:21:50,252] INFO New leader is 1 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

6.测试

创建topic,其中--replication-factor是复制的份数,此数不可以超过节点的个数,--partitions标示分区数,--topic标示名称

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-topics.sh --create --zookeeper single:2181 --replication-factor 1 --partitions 1 --topic mytest

Created topic "mytest".

列出topic

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-topics.sh --list --zookeeper sile:2181

Mytest

查看topic的详细信息

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-topics.sh --describe --zookeeper single:2181

Topic:mytest PartitionCount:1 ReplicationFactor:1 Configs:

Topic: mytest Partition: 0 Leader: 1 Replicas: 1 Isr: 1

生产者,输入数据

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-console-producer.sh --broker-list single:9092 --topic mytest

[2015-01-24 22:40:02,318] WARN Property topic is not valid (kafka.utils.VerifiableProperties)

123

234

789

此处不断的输入,就在consumer不断的接收

消费者,消费数据,这个和producer不是在一个窗口

[[email protected] kafka_2.10-0.8.2-beta]# bin/kafka-console-consumer.sh --zookeeper single:2181 --topic mytest --from-beginning

123

234

789

分布式安装:

分别在三台服务器上安装kafka：

tar zxvf kafka_2.10-0.8.1.tgz

修改每台服务器的config/server.properties

broker.id：唯一，填数字，本文中分别为132/133/134

host.name：唯一，填服务器IP，之前配置时，把中间的‘.‘给忘写了，导致kafka集群形同虚设（基本只有leader机器在起作用），以及一系列莫名其妙的问题，伤啊

zookeeper.connect=192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181

先启动zookeeper服务: bin/zkServer.sh start （本文中zookeeper为独立安装，具体过程在此不细述）

再在每台机器上执行： bin/kafka-server-start.sh config/server.properties

你将会看到如下信息：

[plain] view plain copy print ?

1. <SPAN style="FONT-SIZE: 14px; FONT-FAMILY: Microsoft YaHei">[2014-04-02 02:42:14,220] INFO Verifying properties (kafka.utils.VerifiableProperties)

2. [2014-04-02 02:42:14,281] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)

3. [2014-04-02 02:42:14,281] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)

4. [2014-04-02 02:42:14,281] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)

5. [2014-04-02 02:42:14,282] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)

6. [2014-04-02 02:42:14,282] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)

7. [2014-04-02 02:42:14,282] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)

8. [2014-04-02 02:42:14,282] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)

9. [2014-04-02 02:42:14,283] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)

10. [2014-04-02 02:42:14,283] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)

11. [2014-04-02 02:42:14,283] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)

12. [2014-04-02 02:42:14,283] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)

13. [2014-04-02 02:42:14,284] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)

14. [2014-04-02 02:42:14,284] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)

15. [2014-04-02 02:42:14,284] INFO Property zookeeper.connect is overridden to 192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 (kafka.utils.VerifiableProperties)

16. [2014-04-02 02:42:14,284] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)

17. [2014-04-02 02:42:14,309] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)

18. [2014-04-02 02:42:14,312] INFO [Kafka Server 0], Connecting to zookeeper on 192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 (kafka.server.KafkaServer)

19. [2014-04-02 02:42:14,334] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)

20. [2014-04-02 02:42:14,348] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)

21. [2014-04-02 02:42:14,348] INFO Client environment:host.name=ubuntu (org.apache.zookeeper.ZooKeeper)

22. [2014-04-02 02:42:14,348] INFO Client environment:java.version=1.6.0_37 (org.apache.zookeeper.ZooKeeper)

23. [2014-04-02 02:42:14,348] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper)

24. [2014-04-02 02:42:14,348] INFO Client environment:java.home=/etc/local/jdk1.6.0_37/jre (org.apache.zookeeper.ZooKeeper)

25. [2014-04-02 02:42:14,349] INFO Client environment:java.class.path=:/usr/local/kafka_2.10-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/usr/local/kafka_2.10-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/usr/local/kafka_2.10-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/usr/local/kafka_2.10-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/usr/local/kafka_2.10-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/usr/local/kafka_2.10-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/jopt-simple-3.2.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/kafka_2.10-0.8.1.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/log4j-1.2.15.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/scala-library-2.10.1.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/zkclient-0.3.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/usr/local/kafka_2.10-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.ZooKeeper)

26. [2014-04-02 02:42:14,349] INFO Client environment:java.library.path=/etc/local/jdk1.6.0_37/jre/lib/amd64/server:/etc/local/jdk1.6.0_37/jre/lib/amd64:/etc/local/jdk1.6.0_37/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

27. [2014-04-02 02:42:14,349] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

28. [2014-04-02 02:42:14,349] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

29. [2014-04-02 02:42:14,349] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

30. [2014-04-02 02:42:14,349] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

31. [2014-04-02 02:42:14,349] INFO Client environment:os.version=3.8.0-29-generic (org.apache.zookeeper.ZooKeeper)

32. [2014-04-02 02:42:14,349] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)

33. [2014-04-02 02:42:14,349] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)

34. [2014-04-02 02:42:14,349] INFO Client environment:user.dir=/usr/local/kafka_2.10-0.8.1 (org.apache.zookeeper.ZooKeeper)

35. [2014-04-02 02:42:14,352] INFO Initiating client connection, connectString=192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 sessionTimeout=6000 [email protected] (org.apache.zookeeper.ZooKeeper)

36. [2014-04-02 02:42:14,412] INFO Opening socket connection to server /192.168.40.134:2181 (org.apache.zookeeper.ClientCnxn)

37. [2014-04-02 02:42:14,779] INFO Socket connection established to ubuntu.local/192.168.40.134:2181, initiating session (org.apache.zookeeper.ClientCnxn)

38. [2014-04-02 02:42:14,805] INFO Session establishment complete on server ubuntu.local/192.168.40.134:2181, sessionid = 0x14521d0059e0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)

39. [2014-04-02 02:42:14,811] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)

40. [2014-04-02 02:42:15,045] INFO Log directory ‘/tmp/kafka-logs‘ not found, creating it. (kafka.log.LogManager)

41. [2014-04-02 02:42:15,072] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)

42. [2014-04-02 02:42:15,077] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)

43. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

44. SLF4J: Defaulting to no-operation (NOP) logger implementation

45. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

46. [2014-04-02 02:42:15,170] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)

47. [2014-04-02 02:42:15,176] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)

48. [2014-04-02 02:42:15,310] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

49. [2014-04-02 02:42:15,387] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)

50. [2014-04-02 02:42:15,527] INFO Registered broker 0 at path /brokers/ids/0 with address ubuntu:9092. (kafka.utils.ZkUtils$)

51. [2014-04-02 02:42:15,571] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

52. [2014-04-02 02:42:15,654] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

53. </SPAN>

[2014-04-02 02:42:14,220] INFO Verifying properties (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,281] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,281] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,281] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,282] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,282] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,282] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,282] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,283] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,283] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,283] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,283] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,284] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,284] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,284] INFO Property zookeeper.connect is overridden to 192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,284] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)[2014-04-02 02:42:14,309] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)[2014-04-02 02:42:14,312] INFO [Kafka Server 0], Connecting to zookeeper on 192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 (kafka.server.KafkaServer)[2014-04-02 02:42:14,334] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)[2014-04-02 02:42:14,348] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,348] INFO Client environment:host.name=ubuntu (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,348] INFO Client environment:java.version=1.6.0_37 (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,348] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,348] INFO Client environment:java.home=/etc/local/jdk1.6.0_37/jre (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:java.class.path=:/usr/local/kafka_2.10-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/usr/local/kafka_2.10-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/usr/local/kafka_2.10-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/usr/local/kafka_2.10-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/usr/local/kafka_2.10-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/usr/local/kafka_2.10-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/jopt-simple-3.2.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/kafka_2.10-0.8.1.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/log4j-1.2.15.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/scala-library-2.10.1.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/zkclient-0.3.jar:/usr/local/kafka_2.10-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/usr/local/kafka_2.10-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:java.library.path=/etc/local/jdk1.6.0_37/jre/lib/amd64/server:/etc/local/jdk1.6.0_37/jre/lib/amd64:/etc/local/jdk1.6.0_37/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:os.version=3.8.0-29-generic (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,349] INFO Client environment:user.dir=/usr/local/kafka_2.10-0.8.1 (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,352] INFO Initiating client connection, connectString=192.168.40.134:2181,192.168.40.132:2181,192.168.40.133:2181 sessionTimeout=6000 [email protected] (org.apache.zookeeper.ZooKeeper)[2014-04-02 02:42:14,412] INFO Opening socket connection to server /192.168.40.134:2181 (org.apache.zookeeper.ClientCnxn)[2014-04-02 02:42:14,779] INFO Socket connection established to ubuntu.local/192.168.40.134:2181, initiating session (org.apache.zookeeper.ClientCnxn)[2014-04-02 02:42:14,805] INFO Session establishment complete on server ubuntu.local/192.168.40.134:2181, sessionid = 0x14521d0059e0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)[2014-04-02 02:42:14,811] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)[2014-04-02 02:42:15,045] INFO Log directory ‘/tmp/kafka-logs‘ not found, creating it. (kafka.log.LogManager)[2014-04-02 02:42:15,072] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)[2014-04-02 02:42:15,077] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.[2014-04-02 02:42:15,170] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)[2014-04-02 02:42:15,176] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)[2014-04-02 02:42:15,310] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)[2014-04-02 02:42:15,387] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)[2014-04-02 02:42:15,527] INFO Registered broker 0 at path /brokers/ids/0 with address ubuntu:9092. (kafka.utils.ZkUtils$)[2014-04-02 02:42:15,571] INFO [Kafka Server 0], started (kafka.server.KafkaServer)[2014-04-02 02:42:15,654] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

创建topic：

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic mykafka

查看Topic：

bin/kafka-topics.sh --list --zookeeper localhost:2181

显示Topic：mykafka

查看详细信息：

bin/kafktopics.sh --describe --zookeeper 192.168.40.132:2181

Topic:mykafka PartitionCount:1 ReplicationFactor:3 Configs:

Topic: mykafka Partition: 0 Leader: 133 Replicas: 133,134,132 Isr: 134

发送消息：

bin/kafka-console-producer.sh --broker-list 192.168.40.134:9092 --topic mykafka

23423

bin/kafka-console-producer.sh --brokelist 192.168.40.134:9092 --topic mykafka

4533

如果出现以下信息，则需要下载slftj-nop-1.5.jar，并将其cp至kafka的libs目录下：

[plain] view plain copy print ?

1. <SPAN style="FONT-SIZE: 14px; FONT-FAMILY: Microsoft YaHei">SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

2. SLF4J: Defaulting to no-operation (NOP) logger implementation

3. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. </SPAN>

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

接收消息：

bin/kafka-console-consumer.sh --zookeeper 192.168.40.133:2181 --topic mykafka --from-beginning

sfsa

asdg

sfa

asfda

sadf

23423

4533

时间： 2024-08-26 01:39:14

简洁的kafka单节点和分布式安装文档

简洁的kafka单节点和分布式安装文档的相关文章

Hadoop 单节点 & 伪分布安装手记

k8s Kubernetes v1.10 单节点 kubeadm 快速安装

ORACLE LINUX 6.3 + ORACLE 11.2.0.3 RAC + VBOX安装文档

_00024 尼娜抹微笑伊拉克_云计算ClouderaManager以及CHD5.1.0群集部署安装文档V1.0

Hadoop+Hive+Mysql安装文档

Redhat 5.8 ORACLE 11gR2 RAC安装文档2-grid安装

ovirt3.6安装文档

_00024 妳那伊抹微笑_云计算之ClouderaManager以及CHD5.1.0集群部署安装文档V1.0

mysql安装文档（Linux 官网yum安装版）