logstash收集syslog日志

logstash收集syslog日志
注意:生产用syslog收集日志!!!

编写logstash配置文件

#首先我用rubydebug测试数据
[[email protected] conf.d]# cat syslog.conf
input{
    syslog{
    type => "system-syslog"
    host => "192.168.247.135"
    port => "514"
}
}
output{
    stdout{
    codec => "rubydebug"
}
#检查语法
[[email protected] conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/syslog.conf --configtest
Configuration OK
You have new mail in /var/spool/mail/root
[[email protected] ~]# ss -lntp|grep 514
LISTEN     0      50      ::ffff:192.168.247.135:514                     :::*                   users:(("java",pid=9605,fd=14))
#修改rsyslog配置文件让其能访问
[[email protected] ~]# vim /etc/rsyslog.conf
*.* @@192.168.247.135:514
[[email protected] ~]# systemctl restart rsyslog
[[email protected] ~]#
#运行测试
[[email protected] conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/syslog.conf
Settings: Default filter workers: 1
Logstash startup completed
{
           "message" => "Registered Authentication Agent for unix-process:9680:2638370 (system bus name :1.490 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)\n",
          "@version" => "1",
        "@timestamp" => "2018-07-15T10:08:58.000Z",
              "type" => "system-syslog",
              "host" => "192.168.247.135",
          "priority" => 85,
         "timestamp" => "Jul 15 18:08:58",
         "logsource" => "elk-node1",
           "program" => "polkitd",
               "pid" => "686",
          "severity" => 5,
          "facility" => 10,
    "facility_label" => "security/authorization",
    "severity_label" => "Notice"
}
#添加到elk-log.yml文件
[[email protected] conf.d]# cat elk_log.conf
input {
    file {
      path => "/var/log/messages"
      type => "system"
      start_position => "beginning"
    }
    file {
       path => "/var/log/elasticsearch/hejianlai.log"
       type => "es-error"
       start_position => "beginning"
      codec => multiline {
          pattern => "^\["
          negate => true
          what => "previous"
        }
    }
       file {
       path => "/var/log/nginx/access_json.log"
       codec => json
       start_position => "beginning"
       type => "nginx-log"
    }
    syslog{
    type => "system-syslog"
    host => "192.168.247.135"
    port => "514"
}
}
output {

    if [type] == "system"{
        elasticsearch {
           hosts => ["192.168.247.135:9200"]
           index => "systemlog-%{+YYYY.MM.dd}"
        }
    }

    if [type] == "es-error"{
        elasticsearch {
           hosts => ["192.168.247.135:9200"]
           index => "es-error-%{+YYYY.MM.dd}"
        }
    }
       if [type] == "nginx-log"{
        elasticsearch {
           hosts => ["192.168.247.135:9200"]
           index => "nginx-log-%{+YYYY.MM.dd}"
        }
    }
       if [type] == "system-syslog"{
        elasticsearch {
           hosts => ["192.168.247.135:9200"]
           index => "system-syslog-log-%{+YYYY.MM.dd}"
        }
    }
}

#检查语法
[[email protected] conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf --configtestConfiguration OK
#后台运行
[[email protected] conf.d]# ps aux|grep elk|awk ‘{print $2}‘|xargs kill -9
kill: sending signal to 9780 failed: No such process
You have new mail in /var/spool/mail/root
[[email protected] conf.d]# ps aux|grep elk|awk ‘{print $2}‘
9785
[1]+  Killed                  /opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf  (wd: ~)
(wd now: /etc/logstash/conf.d)
[[email protected] conf.d]# ps aux|grep elk
root       9788  0.0  0.0 112704   972 pts/0    R+   18:18   0:00 grep --color=auto elk
[[email protected] conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf &
[1] 9789
#手动添加日志
[[email protected] conf.d]# logger "you hao"
[[email protected] conf.d]# logger "hello world"
[[email protected] conf.d]# logger "跟我一起学猫叫,一起喵喵喵"

Kibana设置

看hand插件上我们能看到system-syslog索引

Kibana上添加system-syslog索引

完美

原文地址:https://www.cnblogs.com/Dev0ps/p/9314481.html

时间: 2024-10-29 04:55:49

logstash收集syslog日志的相关文章

logstash收集rsyslog日志

(1)rsyslog配置 在192.168.1.31配置 #vim /etc/rsyslog.conf *.* @@192.168.1.32:514 //所有设备名,所有日志级别都发送到192.168.1.32的rsyslog #systemctl restart rsyslog (2)测试rsyslog标准输入输出 1.在192.168.1.32测试rsyslog接收 #vim /etc/logstash/conf.d/rsyslog.conf input { syslog { type =>

ELK日志处理之使用logstash收集log4J日志

介绍一下如何从Java工程中导出log4J日志到Logstash. 一.log4j基础 不能免俗的官方介绍: Log4j 是一个使用 Java 语言编写的,可靠.快速.灵活的日志框架(API),使用 Apache Software License 授权.它被移植到 C.C++.C#.Perl.Python.Ruby 和 Eiffel 语言中. Log4j 是高度可配置的,在运行期使用外部的配置文件对其进行配置.它按照优先级别记录日志,并可将日志信息定向输出到各种介质,比如数据库.文件.控制台.U

logstash 收集windows日志--解决日志不能重命名问题

logstash 在windows中收集日志的时候,会导致写日志的程序不能进行日志的切割(windows提示,文件被占用) 从github上下载useJavatoOpenFile    https://github.com/edwinf/ruby-filewatch/tree/useJavatoOpenFile wget https://github.com/jordansissel/ruby-filewatch/archive/master.zip 解压缩后 使用解压缩工具打开logstash

logstash收集java日志,多行合并成一行

使用codec的multiline插件实现多行匹配,这是一个可以将多行进行合并的插件,而且可以使用what指定将匹配到的行与前面的行合并还是和后面的行合并. 1.java日志收集测试 input { stdin { codec => multiline { pattern => "^\[" //以"["开头进行正则匹配 negate => true //正则匹配成功 what => "previous" //和前面的内容进

Logstash收集nginx日志之使用grok过滤插件解析日志

grok作为一个logstash的过滤插件,支持根据模式解析文本日志行,拆成字段. nginx日志的配置: log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"'; logst

logstash收集tomcat日志

1,日志格式 2015-09-28·09:50:48·[http-bio-80-exec-13]·DEBUG·com.weitoo.server.aspect.LogAspect·-{ip:183.16.4.40,url:http://api.xx.com/server/sc/commodity/getOnlineCommodity,param:{"shopId":1000001,"needCategory":false,"needCommodityTot

Logstash收集nginx日志

1.首先是要在nginx里面配置日志格式化输出 log_format main "$http_x_forwarded_for | $time_local | $request | $status | $body_bytes_sent | $request_body | $content_length | $http_referer | $http_user_agent |" "$http_cookie | $remote_addr | $hostname | $upstrea

使用Logstash收集MongoDB日志并通过Zabbix报警

一 应用场景描述 在有些情况下,仅仅通过Zabbix去监控MongoDB的端口和各种状态还不够,MongoDB的日志监控也是很重要的.例如Mongos连接后端的Shard报SocketException错误等. 二

使用Logstash收集MySQL日志

MySQL