Kafka+Zookeeper+Filebeat+ELK 搭建日志收集系统

ELK

ELK目前主流的一种日志系统,过多的就不多介绍了
Filebeat收集日志,将收集的日志输出到kafka,避免网络问题丢失信息
kafka接收到日志消息后直接消费到Logstash
Logstash将从kafka中的日志发往elasticsearch
Kibana对elasticsearch中的日志数据进行展示

image

环境介绍:

软件版本:
- Centos 7.4
- java 1.8.0_45
- Elasticsearch 6.4.0
- Logstash 6.4.0
- Filebeat 6.4.0
- Kibana 6.4.0
- Kafka 2.12
- Zookeeper 3.4.13

服务器:
- 10.241.0.1  squid(软件分发,集中控制)
- 10.241.0.10 node1
- 10.241.0.11 node2
- 10.241.0.12 node3

部署角色
- elasticsearch: 10.241.0.10(master),10.241.0.11,10.241.0.12
  https://www.elastic.co/cn/products/elasticsearch
  Elasticsearch 允许执行和合并多种类型的搜索 （ 结构化、非结构化、地理位置、度量指标 ）搜索方式

- logstash: 10.241.0.10,10.241.0.11,10.241.0.12
  https://www.elastic.co/cn/products/logstash
  Logstash 支持各种输入选择 ，可以在同一时间从众多常用来源捕捉事件

- filebeat: 10.241.0.10,10.241.0.11,10.241.0.12
  https://www.elastic.co/cn/products/beats/filebeat
  Filebeat 内置的多种模块（auditd、Apache、NGINX、System 和 MySQL）可实现对常见日志格式的一键收集、解析和可视化.

- kibana: 10.241.0.10
  https://www.elastic.co/cn/products/kibana
  Kibana 让您能够可视化 Elasticsearch 中的数据并操作 Elastic Stack

- kafka: 10.241.0.10,10.241.0.11,10.241.0.12
  http://kafka.apache.org/
  Kafka是一种高吞吐量的分布式发布订阅消息系统，它可以处理消费者规模的网站中的所有动作流数据
  kafka集群部署前面的博客: https://www.jianshu.com/p/a9ff97dcfe4e

开始安装部署ELK

1.下载安装包及测试安装包完整性

[[email protected] ~]# cat /etc/hosts
10.241.0.1  squid
10.241.0.10 squid
10.241.0.11 node2
10.241.0.12 node3

[[email protected] ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz.sha512
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz.sha512
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz.sha512
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz.sha512

[[email protected] ~]# yum install perl-Digest-SHA
[[email protected] ~]# shasum -a 512 -c  elasticsearch-6.4.0.tar.gz.sha512
elasticsearch-6.4.0.tar.gz: OK
[[email protected] ~]# shasum -a 512 -c  filebeat-6.4.0-linux-x86_64.tar.gz.sha512
filebeat-6.4.0-linux-x86_64.tar.gz: OK
[[email protected] ~]# shasum -a 512 -c  kibana-6.4.0-linux-x86_64.tar.gz.sha512
kibana-6.4.0-linux-x86_64.tar.gz: OK
[[email protected] ~]# shasum -a 512 -c  logstash-6.4.0.tar.gz.sha512
logstash-6.4.0.tar.gz: OK

2.部署elasticsearch

1) Ansible主机清单
[[email protected] ~]# cat /etc/ansible/hosts
[client]
10.241.0.10 es_master=true
10.241.0.11 es_master=false
10.241.0.12 es_master=false

2) 创建es用户和用户组
[[email protected] ~]# ansible client -m group -a ‘name=elk‘
[[email protected] ~]# ansible client -m user -a ‘name=es group=elk home=/home/es shell=/bin/bash‘

3) 将elasticsearch解压到目标主机
[[email protected] ~]# ansible client -m unarchive -a ‘src=/root/elasticsearch-6.4.0.tar.gz  dest=/usr/local owner=es group=elk‘

4)将准备好的es配置文件模板分发到各个节点
[[email protected] ~]# cat elasticsearch.yml.j2
#集群名称及数据存放位置
cluster.name: my_es_cluster
node.name: es-{{ansible_hostname}}
path.data: /data/elk/es/data
path.logs: /data/elk/es/logs
#允许跨域访问
http.cors.enabled: true
http.cors.allow-origin: "*"
#集群中的角色
node.master: {{es_master}}
node.data: true
#允许访问的地址及传输使用的端口
network.host: 0.0.0.0
transport.tcp.port: 9300
#使用tcp传输压缩
transport.tcp.compress: true
http.port: 9200
#使用单播模式去连接其他节点
discovery.zen.ping.unicast.hosts: ["node1","node2","node3"]

5) 执行ansible,分发配置文件
[[email protected] ~]# ansible client -m template -a ‘src=/root/elasticsearch.yml.j2 dest=/usr/local/elasticsearch-6.4.0/config/elasticsearch.yml owner=es group=elk‘

6) 修改系统允许最大打开的文件句柄数等参数,
[[email protected] ~]# cat change_system_args.sh
#!/bin/bash
if [ "`grep 65536 /etc/security/limits.conf`" = "" ]
then
cat >> /etc/security/limits.conf << EOF
# End of file
* - nofile 1800000
        * soft nproc 65536
        * hard nproc 65536
        * soft nofile 65536
        * hard nofile 65536
EOF
fi

if [ "`grep 655360 /etc/sysctl.conf`" = "" ]
then
echo "vm.max_map_count=655360"  >> /etc/sysctl.conf
fi

7) 通过ansible来执行脚本
[[email protected] ~]# ansible client -m script -a ‘/root/change_system_args.sh‘

8) 重启目标主机,是参数生效(因为目标主机重启 所以ansible连不上)
[[email protected] ~]# ansible client -m shell -a ‘reboot‘
10.241.0.11 | UNREACHABLE! => {
    "changed": false,
    "msg": "SSH Error: data could not be sent to remote host \"10.241.0.11\". Make sure this host can be reached over ssh",
    "unreachable": true
}
10.241.0.12 | UNREACHABLE! => {
    "changed": false,
    "msg": "SSH Error: data could not be sent to remote host \"10.241.0.12\". Make sure this host can be reached over ssh",
    "unreachable": true
}
10.241.0.10 | UNREACHABLE! => {
    "changed": false,
    "msg": "SSH Error: data could not be sent to remote host \"10.241.0.10\". Make sure this host can be reached over ssh",
    "unreachable": true
}

9 )创建elk目录
[[email protected] ~]# ansible client -m file -a ‘name=/data/elk/  state=directory owner=es group=elk‘

10) 启动es
[[email protected] ~]# ansible client -m shell -a ‘su - es -c "/usr/local/elasticsearch-6.4.0/bin/elasticsearch -d"‘ 

10.241.0.11 | SUCCESS | rc=0 >>

10.241.0.10 | SUCCESS | rc=0 >>

10.241.0.12 | SUCCESS | rc=0 >>

11) 查看是否启动
[[email protected] ~]# ansible client -m shell -a ‘ps -ef|grep elasticsearch‘
10.241.0.12 | SUCCESS | rc=0 >>
es        3553     1 19 20:35 ?        00:00:48 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.eFvx2dMC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch-6.4.0 -Des.path.conf=/usr/local/elasticsearch-6.4.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /usr/local/elasticsearch-6.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es        3594  3553  0 20:35 ?        00:00:00 /usr/local/elasticsearch-6.4.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root      3711  3710  0 20:39 ?        00:00:00 /bin/sh -c ps -ef|grep elasticsearch
root      3713  3711  0 20:39 ?        00:00:00 grep elasticsearch

10.241.0.10 | SUCCESS | rc=0 >>
es        4899     1 22 20:35 ?        00:00:54 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.1uRdvBGd -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch-6.4.0 -Des.path.conf=/usr/local/elasticsearch-6.4.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /usr/local/elasticsearch-6.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es        4940  4899  0 20:35 ?        00:00:00 /usr/local/elasticsearch-6.4.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root      5070  5069  0 20:39 ?        00:00:00 /bin/sh -c ps -ef|grep elasticsearch
root      5072  5070  0 20:39 ?        00:00:00 grep elasticsearch

10.241.0.11 | SUCCESS | rc=0 >>
es        3556     1 19 20:35 ?        00:00:47 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.fnAavDi0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/local/elasticsearch-6.4.0 -Des.path.conf=/usr/local/elasticsearch-6.4.0/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /usr/local/elasticsearch-6.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es        3597  3556  0 20:35 ?        00:00:00 /usr/local/elasticsearch-6.4.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root      3710  3709  0 20:39 ?        00:00:00 /bin/sh -c ps -ef|grep elasticsearch
root      3712  3710  0 20:39 ?        00:00:00 grep elasticsearch

12) 查看集群状态
[[email protected] ~]# curl -s http://node1:9200/_nodes/process?pretty |grep -C 5 _nodes
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "my_es_cluster",

3.部署Filebeat

1) 分发安装包到客户机
[[email protected] ~]# ansible client -m unarchive -a ‘src=/root/filebeat-6.4.0-linux-x86_64.tar.gz dest=/usr/local‘

2) 修改安装包名称
[[email protected] ~]# ansible client -m shell -a ‘mv /usr/local/filebeat-6.4.0-linux-x86_64 /usr/local/filebeat-6.4.0‘
10.241.0.12 | SUCCESS | rc=0 >>

10.241.0.11 | SUCCESS | rc=0 >>

10.241.0.10 | SUCCESS | rc=0 >>

3) 修改配置文件
[[email protected] ~]# cat filebeat.yml.j2
filebeat.prospectors:
- type: log
  paths:
    - /var/log/supervisor/kafka

output.kafka:
  enabled: true
  hosts: ["10.241.0.10:9092","10.241.0.11:9092","10.241.0.12:9092"]
  topic: kafka_run_log

##参数解释
enabled 表明这个模块是启动的
host  把filebeat的数据发送到那台kafka上
topic 这个很重要,发送给kafka的topic,若topic不存在,则会自动创建此topic

4) 分发到客户机,并将原来的配置文件备份
[[email protected] ~]# ansible client -m copy -a ‘src=/root/filebeat.yml.j2 dest=/usr/local/filebeat-6.4.0/filebeat.yml backup=yes‘

5) 启动filebeat
[[email protected] ~]# ansible client -m shell -a ‘/usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml &‘
10.241.0.11 | SUCCESS | rc=0 >>

10.241.0.10 | SUCCESS | rc=0 >>

10.241.0.12 | SUCCESS | rc=0 >>

6) 查看filebeat进程
[[email protected] ~]# ansible client -m shell -a ‘ps -ef|grep filebeat| grep -v grep‘
10.241.0.12 | SUCCESS | rc=0 >>
root      4890     1  0 22:50 ?        00:00:00 /usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml

10.241.0.10 | SUCCESS | rc=0 >>
root      6881     1  0 22:50 ?        00:00:00 /usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml

10.241.0.11 | SUCCESS | rc=0 >>
root      4939     1  0 22:50 ?        00:00:00 /usr/local/filebeat-6.4.0/filebeat -c /usr/local/filebeat-6.4.0/filebeat.yml

7) 查看是否有topic创建成功
[[email protected] local]# /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper  10.241.0.10:2181
ConsumerTest
__consumer_offsets
kafka_run_log #filebeat创建的topic
topicTest

4.部署Logstash

1) 解压安装包值目标主机
[[email protected] ~]# ansible client -m unarchive -a ‘src=/root/logstash-6.4.0.tar.gz dest=/usr/local owner=es group=elk‘

2) Logstash配置文件
[[email protected] ~]# cat logstash-kafka.conf.j2
input {
    kafka {
        type => "kafka-logs"
        bootstrap_servers => "10.241.0.10:9092,10.241.0.11:9092,10.241.0.12:9092"
        group_id => "logstash"
        auto_offset_reset => "earliest"
        topics => "kafka_run_log"
        consumer_threads => 5
        decorate_events => true
        }
}

output {
    elasticsearch {
    index => ‘kafka-run-log-%{+YYYY.MM.dd}‘
    hosts => ["10.241.0.10:9200","10.241.0.11:9200","10.241.0.12:9200"]
}

3) 使用ansible推送logstash配置文件到目标主机
[[email protected] ~]# ansible client -m copy -a ‘src=/root/logstash.conf.j2 dest=/usr/local/logstash-6.4.0/config/logstash.conf owner=es group=elk‘

4) 启动Logstash
[[email protected] ~]# ansible client -m shell -a ‘su - es -c "/usr/local/logstash-6.4.0/bin/logstash -f /usr/local/logstash-6.4.0/config/logstash.conf &"‘     

5)_查看Logstash进程
[[email protected] ~]# ansible client -m shell -a ‘ps -ef|grep logstash|grep -v grep‘
10.241.0.11 | SUCCESS | rc=0 >>
es        6040     1 99 23:39 ?        00:02:11 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash-6.4.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash-6.4.0/config/logstash.conf

10.241.0.12 | SUCCESS | rc=0 >>
es        5970     1 99 23:39 ?        00:02:13 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash-6.4.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash-6.4.0/config/logstash.conf

10.241.0.10 | SUCCESS | rc=0 >>
es        9095     1 98 23:39 ?        00:02:10 /usr/local/jdk1.8.0_45/bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash-6.4.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash-6.4.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /usr/local/logstash-6.4.0/config/logstash.conf

5.部署kibana

1) 将安装包拷贝到node1节点
[[email protected] ~]# scp kibana-6.4.0-linux-x86_64.tar.gz [email protected]:/root
kibana-6.4.0-linux-x86_64.tar.gz                 100%  179MB  59.7MB/s   00:03

2) 解压kibana
[[email protected] ~]# tar  -zxf kibana-6.4.0-linux-x86_64.tar.gz  -C /usr/local
[[email protected] ~]# mv /usr/local/kibana-6.4.0-linux-x86_64/ /usr/local/kibana-6.4.0

3) 修改配置文件
[[email protected] ~]# cat /usr/local/kibana-6.4.0/config/kibana.yml
server.port: 5601
server.host: "10.241.0.10"
kibana.index: ".kibana

4) 启动kibana (前台启动)
[[email protected] ~]# /usr/local/kibana-6.4.0/bin/kibana

5) 访问的kibana
http://10.241.0.10:5601

6) 添加日志
Management -> Kibana 列Index Patterns -> Index pattern

7) 发送消息到kafka-run-log  topic,查看是否能通过kibana展示

image

作者：baiyongjie
链接：https://www.jianshu.com/p/d072a55aa844
來源：简书
简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。

原文地址：https://www.cnblogs.com/cheyunhua/p/9765421.html

时间： 2024-10-07 21:31:55

Kafka+Zookeeper+Filebeat+ELK 搭建日志收集系统的相关文章

ELK+kafka构建日志收集系统

ELK+kafka构建日志收集系统原文 http://lx.wxqrcode.com/index.php/post/101.html 背景: 最近线上上了ELK,但是只用了一台Redis在中间作为消息队列,以减轻前端es集群的压力,Redis的集群解决方案暂时没有接触过,并且Redis作为消息队列并不是它的强项:所以最近将Redis换成了专业的消息信息发布订阅系统Kafka, Kafka的更多介绍大家可以看这里: 传送门 ,关于ELK的知识网上有很多的哦, 此篇博客主要是总结一下目前线上这个

2018年ElasticSearch6.2.2教程ELK搭建日志采集分析系统（教程详情）

章节一 2018年 ELK课程计划和效果演示1.课程安排和效果演示简介:课程介绍和主要知识点说明,ES搜索接口演示,部署的ELK项目演示es: localhost:9200kibana http://localhost:5601/ 章节二 elasticSearch 6.2版本基础讲解到阿里云部署实战 2.搜索引擎知识介绍和相关框架简介:介绍搜索的基本概念,市面上主流的搜索框架elasticSearch和solr等对比什么是搜索:在海量信息中获取我们想要的信息传统做法:1.文档中使用系统的Fin

第11天日志收集系统kafka库实战

本节主要内容: 1. 日志收集系统设计2. 日志客户端开发 1. 项目背景 a. 每个系统都有日志,当系统出现问题时,需要通过日志解决问题 b. 当系统机器比较少时,登陆到服务器上查看即可满足 c. 当系统机器规模巨大,登陆到机器上查看几乎不现实 2. 解决方案 a. 把机器上的日志实时收集,统一的存储到中心系统 b. 然后再对这些日志建立索引,通过搜索即可以找到对应日志 c. 通过提供界面友好的web界面,通过web即可以完成日志搜索 3. 面临的问题

用ElasticSearch,LogStash,Kibana搭建实时日志收集系统

用ElasticSearch,LogStash,Kibana搭建实时日志收集系统介绍这套系统,logstash负责收集处理日志文件内容存储到elasticsearch搜索引擎数据库中.kibana负责查询elasticsearch并在web中展示. logstash收集进程收获日志文件内容后,先输出到redis中缓存,还有一logstash处理进程从redis中读出并转存到elasticsearch中,以解决读快写慢速度不一致问题. 官方在线文档:https://www.elastic.co

一共81个，开源大数据处理工具汇总（下），包括日志收集系统/集群管理/RPC等

作者:大数据女神-诺蓝(微信公号:dashujunvshen).本文是36大数据专稿,转载必须标明来源36大数据. 接上一部分:一共81个,开源大数据处理工具汇总(上),第二部分主要收集整理的内容主要有日志收集系统.消息系统.分布式服务.集群管理.RPC.基础设施.搜索引擎.Iaas和监控管理等大数据开源工具. 日志收集系统一.Facebook Scribe 贡献者:Facebook 简介:Scribe是Facebook开源的日志收集系统,在Facebook内部已经得到大量的应用.它能够从各种

[转载] 一共81个，开源大数据处理工具汇总（下），包括日志收集系统/集群管理/RPC等

原文: http://www.36dsj.com/archives/25042 接上一部分:一共81个,开源大数据处理工具汇总(上),第二部分主要收集整理的内容主要有日志收集系统.消息系统.分布式服务.集群管理.RPC.基础设施.搜索引擎.Iaas和监控管理等大数据开源工具. 日志收集系统一.Facebook Scribe 贡献者:Facebook 简介:Scribe是Facebook开源的日志收集系统,在Facebook内部已经得到大量的应用.它能够从各种日志源上收集日志,存储到一个中央存储

用fabric部署维护kle日志收集系统

最近搞了一个logstash kafka elasticsearch kibana 整合部署的日志收集系统.部署参考lagstash + elasticsearch + kibana 3 + kafka 日志管理系统部署 02 上线过程中有一些环节,觉得还是值的大家注意的比如: 1,应用运维和研发人员要讨论一下日志格式的定义, 2,在logstash取日志和消费端logstash消费日志麻.过滤日志的时候怎么要高效,避免服务本身告成系统压力过大,如果每天要处理过亿日志量,性能不注意,哈哈,可以使

基于Flume的美团日志收集系统(一)架构和设计【转】

美团的日志收集系统负责美团的所有业务日志的收集,并分别给Hadoop平台提供离线数据和Storm平台提供实时数据流.美团的日志收集系统基于Flume设计和搭建而成. <基于Flume的美团日志收集系统>将分两部分给读者呈现美团日志收集系统的架构设计和实战经验. 第一部分架构和设计,将主要着眼于日志收集系统整体的架构设计,以及为什么要做这样的设计. 第二部分改进和优化,将主要着眼于实际部署和使用过程中遇到的问题,对Flume做的功能修改和优化等. 1 日志收集系统简介日志收集是大数据的基石.

基于Flume的美团日志收集系统(一)架构和设计

来自:美团技术博客 http://tech.meituan.com/mt-log-system-arch.html 美团的日志收集系统负责美团的所有业务日志的收集,并分别给Hadoop平台提供离线数据和Storm平台提供实时数据流.美团的日志收集系统基于Flume设计和搭建而成. <基于Flume的美团日志收集系统>将分两部分给读者呈现美团日志收集系统的架构设计和实战经验. 第一部分架构和设计,将主要着眼于日志收集系统整体的架构设计,以及为什么要做这样的设计. 第二部分改进和优化,将主要着眼于