一、介绍
Apache Kafka是由Apache软件基金会开发的一个开源消息系统项目，由Scala写成。Kafka最初是由LinkedIn开发，并于2011年初开源。2012年10月从Apache Incubator毕业。该项目的目标是为处理实时数据提供一个统一、高通量、低等待的平台。
二、安装环境
kafka server IP:
10.0.0.25 10.0.0.26
操作系统：
CentOS 6.5 x86_64
须用到的软件包:
jdk-1.7.0_65-fcs.x86_64 java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
Hosts表记录
cat /etc/hosts 10.0.0.25 kafka1 10.0.0.26 kafka2
三、配置kafka的两个节点(kafka1和kafka2)
配置kafka1
1、解压kafka_2.10-0.8.1.1.tgz到kafka1
tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/ cd /usr/local/ mv kafka_2.10-0.8.1.1/ kafka
2、配置kafka1上的/usr/local/kafka/config/server.properties
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/server.properties broker.id=1 port=9000 num.network.threads=2 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/usr/local/kafka/kafka-logs num.partitions=8 log.retention.hours=168 log.segment.bytes=536870912 log.retention.check.interval.ms=60000 log.cleaner.enable=false zookeeper.connect=kafka1:2181,kafka2:2181 zookeeper.connection.timeout.ms=1000000
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeper clientPort=2181 maxClientCnxns=0
配置kafka2
1、解压kafka_2.10-0.8.1.1.tgz到kafka1
tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/ cd /usr/local/ mv kafka_2.10-0.8.1.1/ kafka
2、配置kafka2上的/usr/local/kafka/config/server.properties
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/server.properties broker.id=2 port=9000 num.network.threads=2 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/usr/local/kafka/kafka-logs num.partitions=8 log.retention.hours=168 log.segment.bytes=536870912 log.retention.check.interval.ms=60000 log.cleaner.enable=false zookeeper.connect=kafka1:2181,kafka2:2181 zookeeper.connection.timeout.ms=1000000
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeper clientPort=2181 maxClientCnxns=0
四、编写kafka启动、关闭shell脚本
cat /etc/init.d/kafka
#!/bin/bash
source /etc/profile;
function Stop()
{
ps -ef |grep kafka|grep -v grep |awk ‘{print $2}‘|xargs kill -9
}
function Start()
{
/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &> /dev/null &
sleep 1
/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &> /dev/null &
}
case $1 in
stop)
Stop
;;
start)
Start
;;
*)
echo "Usage: `basename $0` {stop|start}"
;;
esac
用法：
# 启动kafka： /etc/init.d/kafka start # 关闭kafka： /etc/init.d/kafka stop
五、让kafka开机自启动
echo ‘/etc/init.d/kafka start > /dev/null ‘ >> /etc/rc.local
时间： 2024-10-12 11:05:58