日志监控_ElasticStack-0003.Logstash输入插件及实际生产案例应用?

新版插件:

扩展: 如果GitHub上面(https://github.com/logstash-plugins/)发布了扩展插件,可通过./bin/logstash-plugin install <plugin-name>,当然升级也很方便./bin/logstash-plugin update <plugin-name>,如果要安装更新本地已有的插件可通过./bin/logstash-plugin install/update <plugin-path>即可.

注意: 默认./bin/logstash-plugin install/update时是到https://rubygems.org/下载包,速度非常慢,所以强烈推荐手动从https://github.com/logstash-plugins/或https://rubygems.org/下载下来更新

输入插件: https://www.elastic.co/guide/en/logstash/current/input-plugins.html

input {
    udp {
        port => 25826
        workers => 2
        queue_size => 20000
        buffer_size => 1452
        codec => collectd { }
        type => "collectd"
    }
}
output{
    stdout{
        codec => rubydebug
    }
}

生产案例:

yum -y install collectd*
cp -rfp /etc/collectd.conf /etc/collectd.conf_org
vim /etc/collectd.conf
# Collectd基础信息采集配置
Hostname    "xm-server-00001"
FQDNLookup  false
LoadPlugin  df
LoadPlugin  cpu
LoadPlugin  disk
LoadPlugin  memory
LoadPlugin  network
<Plugin network>
    Server "10.2.5.51" "25826"
</Plugin>
LoadPlugin  interface
<Plugin interface>
    Interface "eth0"
    IgnoreSelected false
</Plugin>
Include "/etc/collectd.d"

input {
    file {
        path => ["/xm-workspace/xm-webs/xmcloud/logs/*.log"]
        type => "dss-pubserver"
codec => json
        start_position => "beginning"
    }
}
output{
    stdout{
        codec => rubydebug
    }
}

生产案例:

log_format json ‘{"@timestamp":"$time_iso8601",‘
                 ‘"host":"$server_addr",‘
                 ‘"clientip":"$remote_addr",‘
                 ‘"size":$body_bytes_sent,‘
                 ‘"responsetime":$request_time,‘
                 ‘"upstreamtime":"$upstream_response_time",‘
                 ‘"upstreamhost":"$upstream_addr",‘
                 ‘"http_host":"$host",‘
                 ‘"url":"$uri",‘
                 ‘"xff":"$http_x_forwarded_for",‘
                 ‘"referer":"$http_referer",‘
                 ‘"agent":"$http_user_agent",‘
                 ‘"status":"$status"}‘;

注意: 通常导入原有数据进Elasticsearch前需通过filter/date插件规范默认的@timestamp字段值,file插件并不支持递归监视,如果有需求可以数组形式匹配,支持/path/to/*/*/*/*.log写法,如果需要反复测试可将sincedb_path定义为/dev/null,这样每次重启都会自动从头开始读,还有一点儿需要注意的是Windows由于没有inode概念,所以监听文件不靠谱,推荐使用nxlog作为收集端.

input {
    stdin {
        codec => json
        tags => ["dss-pubserver"]
        type => "stdin"
        add_field => {
            "runenv" => "docker"
        }
    }
}
output{
    stdout{
        codec => rubydebug
    }
}

生产案例:

{"status": "", "content_length": "", "body_bytes_sent": "", "http_x_forwarded_for": "", "request_time": "", "errors": {"text": "res body is invalid protocol format args", "line": 128, "func": "process_msg()", "file": "dsspub-server.lua"}, "request_body": "", "http_referer": "", "level": "", "hostname": "", "request": "", "remote_addr": "", "upstream_response_time": "", "time_local": "", "upstream_addr": ""}

input {
    syslog {
        port => "6514"
    }
}
output{
    stdout{
        codec => rubydebug
    }
}

生产案例:

vim /etc/rsyslog.conf 
*.*                                                     @@127.0.0.1:6514

扩展: 如果实在没法切换到tcp协议,可通过这个小工具来实现异步IO的UDP监听数据输入写入Elasticsearch(https://gist.github.com/chenryn/7c922ac424324ee0d695)

input {
    tcp {
        port => "8888"
        mode => "server"
        ssl_enable => false
    }
}
output{
    stdout{
        codec => rubydebug
    }
}

生产实例:

nc 127.0.0.1 8888 < error.log