一、logstash结合redis收集系统日志
架构图:
环境准备:
A主机:elasticsearch主机 IP地址:192.168.7.100
B主机:logstash主机 IP地址:192.168.7.102
C主机:redis主机 IP地址:192.168.7.103
D主机:logstash主机/nginx主机 IP地址:192.168.7.101
1、安装并配置redis
1、安装并配置redis服务,并启动redis服务
[[email protected] ~]# yum install redis -y [[email protected] ~]# vim /etc/redis.conf bind 0.0.0.0 # 监听本地的所有地址 requirepass 123456 #为了redis安全,设置一个密码 [[email protected] ~]# systemctl restart redis # 启动redis服务
2、在logstash-D主机安装logstash服务
1、先安装JDK、并创建软链接
[[email protected] ~]# cd /usr/local/src [[email protected] src]# ls jdk1.8.0_212 jdk-8u212-linux-x64.tar.gz sonarqube-6.7.7 sonarqube-6.7.7.zip [[email protected] src]# tar xvf jdk-8u212-linux-x64.tar.gz [[email protected] src]# ln -s /usr/local/src/jdk-8u212-linux-x64.tar.gz /usr/local/jdk [[email protected] src]# ln -s /usr/local/jdk/bin/java /usr/bin/
2、配置JDK的环境变量,并使其生效
[[email protected] src]# vim /etc/profile.d/jdk.sh # 设置JDK环境变量 export HISTTIMEFORMAT="%F %T `whoami`" export export LANG="en_US.utf-8" export JAVA_HOME=/usr/local/jdk export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin [[email protected] src]# . /etc/profile.d/jdk.sh # 使JDK环境变量生效
3、安装logstash
[[email protected] src]# yum install logstash-6.8.1.rpm -y [[email protected] ~]# vim /etc/profile.d/logstash.sh # 定义logstash环境变量 export PATH=$PATH:/usr/share/logstash/bin/ [[email protected] ~]# . /etc/profile.d/logstash.sh # 使环境变量生效
4、在/etc/logstash/conf.d目录下创建一个写入到redis日志的文件:redis-es.conf
input { file { path => "/var/log/messages" # 收集档期那logstash日志文件 type => "message-101" # 日志类型 start_position => "beginning" stat_interval => "2" # 间隔2s #codec => "json" } } output { if [type] == "message-101" { redis { host => "192.168.7.103" # 将日志传到103的redis主机 port => "6379" # redis的监听端口号 password => "123456" # redisa的登陆密码 db => "1" # redis的数据库类型,默认是0 key => "linux-7-101-key" #自定义的key data_type => "list" # 数据类型改为list } } }
5、如果logstash服务是以logstash用户启动,将logstash系统日志的权限改为644,否则logstash系统日志无法访问。
[[email protected] conf.d]# vim /etc/systemd/system/logstash.service [Unit] Description=logstash [Service] Type=simple User=root #以root方式启动logstash服务,生产中最好以logstash服务启动 Group=root # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with ‘-‘ makes it try to load, but if the file doesn‘t # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target [[email protected] conf.d]# chmod 644 /var/log/messages # 将系统日志权限进行修改。
6、启动logstash服务
# systemctl start logstash
3、开始测试logstash服务
1、在logstash服务上对/var/log/messages系统日志进行输入信息
[[email protected] src]# echo 11 >> /var/log/messages
2、在redis服务器上查登陆redis客户端查看此时显示的KEYS值
[[email protected] ~]# redis-cli -h 192.168.7.103 192.168.7.103:6379> auth 123456 OK 192.168.7.103:6379> SELECT 1 OK 192.168.7.103:6379[1]> KEYS * 1) "linux-7-101-key" # 可以看到此时的logstash服务将logstash服务器的系统日志已经传递到redis服务器上
此时在第二台logstash主机上可以将系统日志传到redis主机上。
4、在logstash-B主机上配置
1、在/etc/logstash/conf.d目录下创建一个提取redis缓存日志文件
input { redis { host => "192.168.7.103" # redis主机IP地址 port => "6379" password => "123456" db => "1" key => "linux-7-101-key" # 取出对应的KEY值 data_type => "list" } } output { if [type] == "message-101" { # 提取与第二台logstash主机的log类型一致 elasticsearch { hosts => ["192.168.7.100:9200"] # elasticsearch主机的IP地址 index => "message-7-101-%{+YYYY.MM.dd}" } } }
2、重启B主机的logstash服务
# systemctl restart logstash
3、此时在redis服务器上查看数据已经被logstash服务器采集到并传到了elasticsearch服务器上。
192.168.7.103:6379[1]> KEYS * (empty list or set) # 此时的redis服务器数据为空
5、在kibana控制台创建索引
1、创建收集到redis数据的索引
2、在discover选项中查看收集到的信息
二、logstash结合redis收集nginx访问日志
1、在D主机安装nginx服务并将log日志配置为json格式
1、安装nginx服务,最好是源码编译,方便后期升级nginx版本
[[email protected] ~]# cd /usr/local/src [[email protected] src]# wget http://nginx.org/download/nginx-1.14.2.tar.gz [[email protected] src]# tar xvf nginx-1.14.2.tar.gz [[email protected] nginx-1.14.2]# ./configure --prefix=/apps/nginx # 安装nginx,并制定安装目录 [[email protected] nginx-1.14.2]# make -j 2 && make install # 编译安装nginx
2、修改nginx配置文件,并将log日志改为json格式/apps/nginx/conf/nginx.conf
log_format access_json ‘{"@timestamp":"$time_iso8601",‘ ‘"host":"$server_addr",‘ ‘"clientip":"$remote_addr",‘ ‘"size":$body_bytes_sent,‘ ‘"responsetime":$request_time,‘ ‘"upstreamtime":"$upstream_response_time",‘ ‘"upstreamhost":"$upstream_addr",‘ ‘"http_host":"$host",‘ ‘"url":"$uri",‘ ‘"domain":"$host",‘ ‘"http_user_agent":"$http_user_agent",‘ ‘"xff":"$http_x_forwarded_for",‘ ‘"referer":"$http_referer",‘ ‘"status":"$status"}‘; access_log /var/log/nginx/access.log access_json; # 定义json格式的日志,并指定存放在/var/log/nginx目录下 [[email protected] nginx-1.14.2]# mkdir /var/log/nginx # 创建一个存放log日志的目录
3、启动nginx服务,并查看启动的80端口
[[email protected] nginx-1.14.2]# /apps/nginx/sbin/nginx [[email protected] nginx-1.14.2]# ss -nlt State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 511 *:80 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 [::1]:25 [::]:* LISTEN 0 50 [::ffff:127.0.0.1]:9600 [::]:* LISTEN 0 128 [::]:22 [::]:*
2、修改第logstash-D主机的配置文件
1、在第二台logstash主机的/etc/logstash/conf.d目录下创建配置文件
input { file { path => "/var/log/messages" type => "message-7-101" start_position => "beginning" stat_interval => "2" #codec => "json" } file { path => "/var/log/nginx/access.log" type => "nginx-accesslog-7-101" start_position => "beginning" stat_interval => "2" codec => "json" } } output { if [type] == "message-7-101" { redis { host => "192.168.7.103" port => "6379" password => "123456" db => "1" key => "linux-7-101-key" data_type => "list" }} if [type] == "nginx-accesslog-7-101" { redis { host => "192.168.7.103" port => "6379" password => "123456" db => "1" key => "linux-nginxlog-7-101-key" data_type => "list" } } }
2、重启logstash服务,将第一台的logstash主机服务停掉。
[[email protected] conf.d]# systemctl restart logstash
3、在redis主机上查看logstash主机是否已经将数据传到redis上
192.168.7.103:6379[1]> KEYS * 1) "linux-7-101-key" # 系统日志 2) "linux-nginxlog-7-101-key" # nginx日志
3、在logstash-A主机上修改配置文件
1、在/etc/logstash/conf.d目录下创建一个提取redis数据的配置文件
input { redis { host => "192.168.7.103" port => "6379" password => "123456" db => "1" key => "linux-7-101-key" # 与第二台的logstash服务器key对应 data_type => "list" } redis { host => "192.168.7.103" port => "6379" password => "123456" db => "1" key => "linux-nginxlog-7-101-key" # 与第二台logstash服务器对应 data_type => "list" } } output { if [type] == "message-7-101" { # 与第二台logstash服务器对应 elasticsearch { hosts => ["192.168.7.100:9200"] index => "message-7-101-%{+YYYY.MM.dd}" } } if [type] == "nginx-accesslog-7-101" { # 与第二台logstash服务器对应 elasticsearch { hosts => ["192.168.7.100:9200"] index => "nginx-accesslog-7-101-%{+YYYY.MM.dd}" } } }
2、重启logstash服务
# systemctl restart logstash
3、查看reids主机的数据,此时数据已经为空,被此台logstash服务器已经取走
192.168.7.103:6379[1]> KEYS * (empty list or set) 192.168.7.103:6379[1]> KEYS * (empty list or set)
4、在kibaba网页上创建索引
1、在kibana网页上创建索引
2、查看discover选项添加的索引信息
原文地址:https://www.cnblogs.com/struggle-1216/p/12491620.html