Logstash管道可以配置一个或多个输入插件、过滤器插件和输出插件。其中，输入插件和输出插件是必选的，过滤器插件是可选的。下图是Logstash常见的使用场景。
上一节的例子中我们使用标准的输入和输出插件做了简单的示例。接下来我们演示一些复杂的场景。如下图所示是Logstash的标准管道结构，我们通过一些高级配置来完成Apache日志的过滤。
|
# The # character at the beginning of a line indicates a comment. Use# comments to describe your configuration. input { } # The filter part of this file is commented out to indicate that it is# optional. # filter { # # } output { } |
1. 准备一段apache日志文件，格式如下：
|
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" |
2. 编写Logstash管道配置文件，放在Logstash/bin目录下
input {
file {
path => "/opt/cx/logstash/apache-log.log"
start_position => beginning
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {}
stdout {}
}
3. 校验配置文件是否正确
[[email protected] bin]# ./logstash -f apache-log-pipeline.conf --configtest Configuration OK
4.启动Logstash
[[email protected] bin]# ./logstash -f apache-log-pipeline.conf
Settings: Default pipeline workers: 4
Pipeline main started
5.完整的Logstash配置文件如下
input {
file {
path => "/opt/cx/logstash/apache-log.log"
start_position => beginning
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts=>["10.0.10.5:9200"]
}
stdout {}
}