利用ELK+redis搭建一套nginx日志分析平台。
使用Elasticsearch + Logstash + Kibana+redis来搭建nginx日志监控分析。
nginx的每个请求状态等都有日志文件来记录，可以通过读取日志文件来分析；
logstash有agent（Shipper）和server（Indexer)端，agent端负责监控本地日志文件的变化，
及时把日志文件的最新内容收集起来，输出到Redis暂存；
redis的list结构正好可以作为队列使用，用来存储logstash传输的日志数据；
server端：日志存储，负责从Redis接收日志；
分析之后存储到elasticsearch进行搜索分析再由统一的kibana进行日志web界面的展示.
实现所需架构图：
部署3台服务器：
1.192.168.0.156：安装nginx+logstash+redis:负责监控本地日志文件的变化，
及时把日志文件的最新内容收集起来，输出到Redis暂存
2.192.168.0.155：安装logstash,作为server端，日志存储者，负责从Redis接收日志
3.192.168.0.188：安装elasticsearch+kibana,搜索分析再由统一的kibana进行日志web界面的展示
实现步骤：
- 首先在192.168.0.156安装nginx+logstash+redis:（提前下载好logstash1.5.4）
[[email protected] ~]# yum install nginx -y
[[email protected] ~]# systemctl startnginx.service ###启动nginx
[[email protected] ~]# yum install redis -y
[[email protected] ~]# vim /etc/redis.conf
bind 0.0.0.0 ###监听本机所有地址
[[email protected] ~]# systemctl start redis.service ###启动redis
[[email protected] ~]# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel –y
###安装jdk
[[email protected] dylan]# yum localinstalllogstash-1.5.4-1.noarch.rpm –y
###安装logstash1.5.4
[[email protected] dylan]# vim/etc/profile.d/logstash.sh
export PATH=/opt/logstash/bin:$PATH
[[email protected] dylan]# ./etc/profile.d/logstash.sh
[[email protected] dylan]# cd/etc/logstash/conf.d/
[[email protected] conf.d]# vim/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core
-0.3.0/patterns/grok-patterns ###ngigx_log的匹配方式，添加以下日志格式
#NGINX
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} -%{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(%{WORD:verb}%{NOTSPACE:request}(?:HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response}(?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}%{NOTSPACE:http_x_forwarded_for}
[[email protected] conf.d]# vimnglogredissample.conf ###配置logstash收集nginx日志
input {
file {
path =>["/var/log/nginx/access.log"]
type => "nginxlog"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{NGINXACCESS}"} }
}
output {
redis { ###日志发往redis暂存
port => "6379"
host =>["127.0.0.1"]
data_type =>"list"
key =>"logstash-%{type}"
}
}
[[email protected] conf.d]# logstash -f./nglogredissample.conf –configtest ###测试配置文件
Configuration OK
[[email protected] conf.d]# logstash -f./nglogredissample.conf ###启用配置文件
Logstash startup completed
[[email protected] /]# redis-cli ###连接至redis
127.0.0.1:6379> LINDEX logstash-nginxlog1 ###可通过redis查看
- 在192.168.0.155安装logstash,作为server端，日志存储者，负责从Redis接收日志
[[email protected] ~]# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel –y
###安装jdk
[[email protected] dylan]# yum localinstalllogstash-1.5.4-1.noarch.rpm –y
###安装logstash1.5.4
[[email protected] dylan] vim/etc/profile.d/logstash.sh
export PATH=/opt/logstash/bin:$PATH
[[email protected] dylan]# ./etc/profile.d/logstash.sh
[[email protected] dylan]# cd/etc/logstash/conf.d/
[[email protected] conf.d]# vim server.conf ###配置logstash server端
input {
redis { ###从redis获取数据
port => "6379" ###redis端口
host =>"192.168.0.156" ###redis主机
data_type =>"list"
key => "logstash-nginxlog"
}
}
output {
elasticsearch{ ###输出至elasticsearch
cluster =>"logstash" ###elasticsearch中的cluster
index => "logstash-%{+YYYY.MM.dd}" ###索引格式
}
}
[[email protected] conf.d]# logstash -f./server.conf –configtest ###测试语法
[[email protected] conf.d]# logstash -f./server.conf ###运行server配置
- 在192.168.0.188：安装elasticsearch+kibana, 搜索分析再由统一的kibana进行日志web界面的展示。
提前下载准备好：
elasticsearch-1.7.2.noarch.rpm
bigdesk-latest.zip （一个ES插件扩展，可查看es集群的各种状态）
kibana-4.1.2-linux-x64.tar.gz
[[email protected] ~]# yum install java-1.8.0-openjdkjava-1.8.0-openjdk-devel –y
###安装jdk
[[email protected] dylan]# yun installelasticsearch-1.7.2.noarch.rpm –y ###安装es1.7.2
[[email protected] elasticsearch]# vim/etc/elasticsearch/elasticsearch.yml
cluster.name: logstash ###配置cluster
node.name: "node1.xiao.com"
[[email protected] elasticsearch]# systemctldaemon-reload
[[email protected] elasticsearch]# systemctl startelasticsearch.service ###启动es
es节点之间交互的tcp端口，默认是9300
对外服务的http端口，默认为9200
[[email protected] ~]#/usr/share/elasticsearch/bin/plugin -i bigdesk -ufile:///dylan/plugins/bigdesk-latest.zip ###安装bigdesk插件
http://192.168.0.188:9200/_plugin/bigdesk ###网页端访问可查看
[[email protected] dylan]# tar xfkibana-4.1.2-linux-x64.tar.gz -C /usr/local ###解压kibana
[[email protected] dylan]# cd /usr/local/
[[email protected] local]# ln -svkibana-4.1.2-linux-x64 kibana ###创建链接
"kibana" ->"kibana-4.1.2-linux-x64"
[[email protected] local]# cd kibana/config/
[[email protected] config]# vim kibana.yml ###编辑配置文件
elasticsearch_url:"http://localhost:9200" ###同属同一台服务器
[[email protected] config]# cd ..
[[email protected] kibana]# bin/kibana ###启动kibana
网页端输入：http://192.168.0.188:5601 查看
测试：在网页端刷新nginx,产生日志，即可通过kibana查看。
