Spark修炼之道（进阶篇）——Spark入门到精通：第十五节 Kafka 0.8.2.1 集群搭建

作者：周志湖

微信号：zhouzhihubeyond

本节为下一节Kafka与Spark Streaming做铺垫

主要内容

1.kafka 集群搭建

1. kafka 集群搭建

1. kafka 安装与配置

   到下面的地址下载：Scala 2.10 - kafka_2.10-0.8.2.1.tgz

   http://kafka.apache.org/downloads.html

   下载完成后，使用命令

tar -zxvf  kafka_2.10-0.8.2.1.tgz 

解压，解压后的目录如下

进入config目录，将server.properties文件内容如下：

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkmaster

//中间省略，默认配置即可
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

将整个安装文件进行跨机器拷贝：

root@sparkmaster:/hadoopLearning# scp -r kafka_2.10-0.8.2.1/ sparkslave01:/hadoopLearning/
root@sparkmaster:/hadoopLearning# scp -r kafka_2.10-0.8.2.1/ sparkslave02:/hadoopLearning/ 

将sparkslave01机器上的server.properties文件内容如下：

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkslave01

//中间省略，默认配置即可

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

将sparkslave02机器上的server.properties文件内容如下：

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkslave02

//中间省略，默认配置即可

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

1. 启动Kafka集群

root@sparkslave02:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties
root@sparkslave01:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties
root@sparkmaster:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 

3 创建topic

在sparkmaster机器上执行下列命令创建一个topic

root@sparkmaster:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-topics.sh --create --topic kafkatopictest --replication-factor 3 --partitions 2 --zookeeper sparkmaster:2181
Created topic "kafkatopictest".

4 发送消息至kafka

在sparkslave01机器上执行下列命令并向kafka发送消息

root@sparkslave01:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-console-producer.sh --broker-list sparkslave01:9092 --sync --topic kafkatopictest
Hello Kafka, I will test Spark Streaming on you next lesson

5 接收kafka发送来的消息

在sparkslave02机器上执行下列命令并接收kafka发送消息

root@sparkslave02:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-console-consumer.sh --zookeeper sparkmaster:2181 --topic kafkatopictest --from-beginning
Hello Kafka, I will test Spark Streaming on you next lesson

至此Kafka 集群搭建与测试完毕

下一节当中，我们将演示kafka如何与Spark Streaimg结合起来使用