Install and Configure Apache Kafka on Ubuntu 16.04

https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/

by hitjethvaon Oct 03, 2016

Intermediate

Table of Contents

Introduction

Apache Kafka is an open-source scalable and high-throughput messaging system developed by the Apache Software Foundation written in Scala. Apache Kafka is specially designed to allow a single cluster to serve as the central data backbone for a large environment. It has a much higher throughput compared to other message brokers systems like ActiveMQ and RabbitMQ. It is capable of handling large volumes of real-time data efficiently. You can deploy Kafka on single Apache server or in a distributed clustered environment.

Features

The general features of Kafka are as follows :

  1. Persist message on disk that provide constant time performance.
  2. High throughput with disk structures that supporting hundreds of thousands of messages per second.
  3. Distributed system scales easily with no downtime.
  4. Supports multi-subscribers and automatically balances the consumers during failure.

This tutorial shows how to install and configure Apache Kafka on a Ubuntu 16.04 server.

Requirements

  • A Ubuntu 16.04 server.
  • Non-root user account with sudo privilege set up on your server.

Getting Started

Let‘s start making sure that your Ubuntu 16.04 server is fully up to date.

You can update your server by running the following command:

sudo apt-get update -y
sudo apt-get upgrade -y

Installing Java

Before installing Kafka, you will need to install Java on your system. You can install Oracle JDK 8 using the Webupd8 team PPA repository.

To add the repository, run the following command:

sudo add-apt-repository -y ppa:webupd8team/java

You should see the following output:

gpg: keyring `/tmp/tmpkjrm4mnm/secring.gpg‘ created
gpg: keyring `/tmp/tmpkjrm4mnm/pubring.gpg‘ created
gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpkjrm4mnm/trustdb.gpg: trustdb created
gpg: key EEA14886: public key "Launchpad VLC" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

Next, update the metadata of the new repository by running the following command:

sudo apt-get update

Once you have finished, run the following command to install JDK 8:

sudo apt-get install oracle-java8-installer -y

You can also verify that JDK 8 is installed properly by running the following command:

sudo java -version

You should see the output something like this:

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

Install ZooKeeper

Before installing Apache Kafka, you will need to have zookeeper available and running. ZooKeeper is an open source service for maintaining configuration information, providing distributed synchronization, naming and providing group services.

By default ZooKeeper package is available in Ubuntu‘s default repository, you can install it by running the following command:

sudo apt-get install zookeeperd

Once installation is finished, it will be started as a daemon automatically. By default ZooKeeper will run on port 2181.

You can test it by running the following command:

netstat -ant | grep :2181

If everything‘s fine, you should see the following Output:

tcp6       0      0 :::2181                 :::*                    LISTEN

Install and Start Kafka Server

Now that Java and ZooKeeper are installed, it is time to download and extract Kafka from Apache website. You can use wget to download Kafka:

wget http://mirror.fibergrid.in/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz

Next, create a directory for Kafka installation:

sudo mkdir /opt/Kafka
cd /opt/Kafka

Extract the downloaded archive using tar command in /opt/Kafka:

sudo tar -xvf kafka_2.10-0.10.0.1.tgz -C /opt/Kafka/

The next step is to start Kafka server, you can start it by running kafka-server-start.sh script located at /opt/Kafka/kafka_2.10-0.10.0.1/bin/ directory.

sudo  /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/server.properties

You should see the following output, if the server has started successfully:

[2016-08-22 21:43:48,279] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2016-08-22 21:43:48,516] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2016-08-22 21:43:48,525] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2016-08-22 21:43:48,527] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-08-22 21:43:48,555] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

You can use nohup with script to start the Kafka server as a background process:

sudo nohup /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/server.properties /tmp/kafka.log 2>&1 &

You now have a Kafka server running and listening on port 9092.

Testing Kafka Server

Now, it is time to verify the Kafka server is operating correctly.

To test Kafka, create a sample topic with name "testing" in Apache Kafka using the following command:

sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1  --partitions 1 --topic testing

You should see the following output:

Created topic "testing".

Now, ask Zookeeper to list available topics on Apache Kafka by running the following command:

sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181

You should see the following output:

testing

Now, publish a sample messages to Apache Kafka topic called testing by using the following producer command:

sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testing

After running above command, enter some messages like "Hi how are you?" press enter, then enter another message like "Where are you?"

Now, use consumer command to check for messages on Apache Kafka Topic called testing by running the following command:

sudo /opt/Kafka/kafka_2.10-0.10.0.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testing --from-beginning

You should see the following output:

Hi how are you?
Where are you?

With this above testing you have successfully verified that you have a valid Apache Kafka setup with Apache Zookeeper.

Summary

At this point, we have installed, configured, and tested Kafka on a Ubuntu 16.04 server. You can adapt the setup to make use of it in your production environment. To learn more about Kafka check out the Kafka documentation.

原文地址:https://www.cnblogs.com/xiaobin-hlj80/p/8998181.html

时间: 2024-08-04 14:39:12

Install and Configure Apache Kafka on Ubuntu 16.04的相关文章

Install .NET Core Runtime on Linux Ubuntu 16.04 x64

原文链接https://www.microsoft.com/net/download/linux-package-manager/ubuntu16-04/runtime-current nstall .NET Core Runtime on Linux Ubuntu 16.04 x64 Not sure where to start? See Get started with .NET in 10 minutes. Linux Distribution                  RHEL

ubuntu 16.04源码编译和配置caffe详细教程 | Install and Configure Caffe on ubuntu 16.04

本文首发于个人博客https://kezunlin.me/post/b90033a9/,欢迎阅读! Install and Configure Caffe on ubuntu 16.04 Series Part 1: Install and Configure Caffe on windows 10 Part 2: Install and Configure Caffe on ubuntu 16.04 Guide requirements: NVIDIA driver 396.54 CUDA 8

Install LAMP Stack On Ubuntu 16.04

原文:http://www.unixmen.com/how-to-install-lamp-stack-on-ubuntu-16-04/ LAMP is a combination of operating system and open-source software stack. The acronym LAMP came from the first letters ofLinux, Apache HTTP Server, MySQL or MariaDB database, and PH

Ubuntu 16.04环境布署小记

本系列文章记录了升级Ubuntu 16.04的布署过程 回到目录 10. 安装Mono, Xsp 当前版本16.04.1的系统源的Mono版本为4.2.1,如需使用最新版本(本文书写时稳定版本为4.6.2),则需添加Mono官方源地址 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo "deb http://dow

Ubuntu 16.04 LTS 成功编译 Android 6.0 源码教程 (转)

1 前言 经过3天奋战,终于在Ubuntu 16.04上把Android 6.0的源码编译出来了,各种配置,各种error,各种爬坑,特写此博客记录爬坑经历.先上图,Ubuntu上编译完后成功运行模拟器,如图: 2 编译环境 UbuntuKylin 16.04 LTS Android 6.0_r1 Open JDK 7 3 准备工作 (1) 下载Android 6.0源码. Androdi 6.0源码下载地址: http://pan.baidu.com/s/1o6N86a2 感谢下面这位博主上传

Installation Guide Ubuntu 16.04

Beside the installation guide on the main page, here is a guide to install GenieACS off a freshly installed Ubuntu 16.04 LTS. Prequisites and basic components First you should update all components and install the following perquisites sudo apt-get u

mariadb install on Ubuntu 16.04.1 LTS

最近在尝试基于Ubuntu 16.04.1 LTS server 部署openstack ,在部署的过程中遇到mariadb 无法正常链接. 问题 参照openstack 官方文档安装mariadb,安装后遇到无法在本地登录的问题 $mysql -u root -p $mysql -h localhost -u root -p 报错如下 mysql (mariadb) ERROR 1698 (28000): Access denied for user 'root'@'localhost' 但使

How To Install Java with Apt-Get on Ubuntu 16.04

Koen Vlaswinkel Subscribe Share 29 How To Install Java with Apt-Get on Ubuntu 16.04 PostedApril 23, 2016 1.1mviews JAVA UBUNTU UBUNTU 16.04 Introduction Java and the JVM (Java's virtual machine) are widely used and required for many kinds of software

在Ubuntu 16.04上使用Apache安装phpBB

PhpBB是一个开源的公告板程序.本文将向您展示如何在Ubuntu 16.04上在Apache webserver上安装phpBB.它是使用phpBB 3.2.1编写的,但是这里提供的说明可能也适用于新版本的phpBB.先决条件本文假设您已经使用Ubuntu 16.04创建了一个Vultr云计算实例,并以root身份登录. 步骤1:安装Apache更新存储库列表.apt-get update安装Apache web服务器.apt-get install apache2步骤2:安装MySQLphp