CentOS 7环境下Kafka的安装和基本使用

CentOS 7环境下Kafka的安装和基本使用

基础环境

Windows 10 X64

VMware-workstation-full-12.0.0-2985596

CentOS Linux release 7.4.1708

CentOS 7环境下安装Kafka

安装JDK

• 在CentOS上将JDK下载到本地.jdk-8u152-linux-x64.tar.gz
• 解压并将文件移动到/opt/java目录下

sudo tar -vxzf jdk-8u152-linux-x64.tar.gzsudo mv jdk1.8.0_152 /opt/java

• 配置java环境变量

vim /etc/profile#在profile文件的最后面添加export JAVA_HOME=/opt/jdk1.8.0_152export PATH=$JAVA_HOME/bin:$PATH#添加成功后通过java -version来查看jdk是否安装成功java -versionopenjdk version "1.8.0_131"OpenJDK Runtime Environment (build 1.8.0_131-b12)OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

安装Zookeeper

• 通过wget下载zookeeper到本机

sudo wdget https://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

• 解压并将文件移动到/opt/zookeeper目录下

# 解压zookeepersudo tar -zxvf zookeeper-3.3.6.tar.gz# 将zookeeper移动到/opt/zookeeper目录下sudo mv zookeeper-3.3.6 /opt/zookeeper

• 编辑zookeeper的配置

# 复制一份zoo_sample.cfg文件并改名为zoo.cfgsudo cp /opt/zookeeper/zoo_sample.cfg zoo.cfg# 编辑zoo.cfg 文件sudo vim /opt/zookeeper/zoo.cfg#主要修改dataDir和server.1=127.0.0.1:2888:3888这2处# the directory where the snapshot is stored.dataDir=/opt/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=127.0.0.1:2888:3888

• 配置zookeeper环境变量

sudo vim /etc/profile#添加如下内容ZOOKEEPER_HOME=/opt/zookeeperPATH=$PATH:ZOOKEEPER_HOME/binexport ZOOKEEPER_HOME

• 启动zookeeper

zkServer.sh start# 如果上面命令报错,则cd /opt/zookeeper/bin 目录./zkServer.sh start

安装kafka

• 通过wget下载kafka到本机

sudo wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.0.0/kafka_2.12-1.0.0.tgz

• 解压并将文件移动到/opt/kafka目录下

# 解压sudo tar -zxvf kafka_2.12-1.0.0.tgz# 移动sudo mv kafka_2.12-1 /opt/kafka

• 编辑kafka的配置

#创建日志存放目录cd /opt/kafkamkdir -p log/kafka#修改配置文件/opt/kafka/config/server.propertiessudo vim /opt/kafka/config/server.properties#主要修改下面几项内容如下：broker.id=181delete.topic.enable=truelisteners = PLAINTEXT://127.0.0.1:9092log.dirs=/opt/kafka/log/kafkazookeeper.connect=127.0.0.1:2181

• 配置kafka环境变量

sudo vim /etc/profile#添加如下内容：KAFKA_HOME=/opt/kafkaPATH=$PATH:$KAFKA_HOME/binexport PATH KAFKA_HOME

• 启动kafka

kafka-server-start.sh /opt/kafka/config/server.properties 

CentOS 7环境下Kafka的基本使用

• 创建一个Topic (test)

kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test

• Producer向Topic发送消息

kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test

[[email protected] bin]$ ./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic testOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N>producer send one message>producer send two message>         

• Consumer读取Topic的消息

kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test --from-beginning

[[email protected] bin]$ ./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 -topic test --from-beginningOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=NUsing the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].[2017-11-19 20:32:13,047] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)[2017-11-19 20:32:13,085] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)[2017-11-19 20:32:13,100] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)[2017-11-19 20:32:13,265] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)producer send one messageproducer send two message

参考资料

CentOS 搭建JDK环境

CentOS7安装配置ZooKeeper集群

CentOS7.0安装配置Kafka集群

kafka环境搭建1-单机版

Confluent.Kafka (Kafka .NET client)