我所期望达到的效果,就是每个日期一条,在kibana界面也是一条。多行异常的也是一条。
其实,很简单,就是加个反向判断。
logstash原理
一个客户端,一个服务器,就是这样的模式
没什么神奇的,最麻烦的正则匹配,不好弄。听说storm更好用。
1.client配置
cat /etc/logstash/conf.d/shipper.conf
input { file { path => ["/opt/src/logs/*/*/*/*"] type => "service" start_position => "beginning" } } filter { if [type] == "service" { multiline { patterns_dir => "/etc/logstash/conf.d" pattern => "(^%{MYTIMESTAMP})" negate => true what => "previous" } } grok { patterns_dir => "/etc/logstash/conf.d" match => [ "message", "%{MYLOG}" ] add_field => [ "log_ip", "随便写" ] } } output { stdout {} redis { host => "你的服务端ip" port => 6379 password => "8ff35947f8efe8db806622f6a98a1ea3" data_type => "list" key => "key_count" } }
patterns_dir => "/etc/logstash/conf.d" 这个东西就是正则表达式
新建一个文件 cat /etc/logstash/conf.d/j2ee
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]* HTTPPORT ([a-zA-Z$_0-9]*\-)*([0-9]) JAVALOGMESSAGE (.*) MYTIMESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{HOUR}:%{MINUTE}:%{SECOND} MYLOG %{MYTIMESTAMP:mytimestamp}\s\[%{HTTPPORT:port}\]\s%{LOGLEVEL:level}\s%{JAVACLASS:class}\s-%{JAVALOGMESSAGE:logmessage}
我的日志格式是这样的
2016-10-2015:52:01.174 [http-apr-8282-exec-4] INFO c.e.w.c.WorkFlowRestController -resp:
如果你的日志不一样,那么j2ee文件要做相应的修改
http://grokdebug.herokuapp.com/ 这个网站可以测试
比如
启动脚本logstash.sh
#!/bin/bash . /etc/init.d/functions function start(){ cd /home/ nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/shipper.conf & >>/dev/null 2>&1 } function stop(){ kill -9 `ps -ef|grep logstash|grep -v grep|awk ‘{print $2}‘` } case $1 in start) start ;; stop) stop ;; *) printf "Usage sh *.sh start|stop" ;; esac
sh logstash.sh stop/start
2.server配置
cat /etc/logstash/conf.d/indexer.conf
input { redis { host => "172.29.12.193" port => 6379 password => "8ff35947f8efe8db806622f6a98a1ea3" type => "redis-input" data_type => "list" key => "key_count" } } output { stdout {} elasticsearch { cluster => "elasticsearch" codec => "json" protocol => "http" } }
3.kibana效果
多行为一行就不截图了
时间: 2024-10-13 22:26:59