logstash作为数据搜集器，主要分为三个部分：input->filter->output 作为pipeline的形式进行处理,支持复杂的操作，如发邮件等
input配置数据的输入和简单的数据转换
filter配置数据的提取，一般使用grok
output配置数据的输出和简单的数据转换
运行：logstash -f /etc/logstash.conf
-f 指定配置文件
-e 只在控制台运行
具体的配置见官网
https://www.elastic.co/products/logstash
Centralize, Transform & Stash Your Data
input
|
Plugin |
Description |
Github repository |
|
beats |
Receives events from the Elastic Beats framework |
logstash-input-beats |
|
couchdb_changes |
Streams events from CouchDB’s _changes URI
|
logstash-input-couchdb_changes |
|
elasticsearch |
Reads query results from an Elasticsearch cluster |
logstash-input-elasticsearch |
|
file |
Streams events from files |
logstash-input-file |
|
gelf |
Reads GELF-format messages from Graylog2 as events |
logstash-input-gelf |
|
generator |
Generates random log events for test purposes |
logstash-input-generator |
|
graphite |
Reads metrics from the graphite tool
|
logstash-input-graphite |
|
heartbeat |
Generates heartbeat events for testing |
logstash-input-heartbeat |
|
http |
Receives events over HTTP or HTTPS |
logstash-input-http |
|
http_poller |
Decodes the output of an HTTP API into events |
logstash-input-http_poller |
|
jdbc |
Creates events from JDBC data |
logstash-input-jdbc |
|
kafka |
Reads events from a Kafka topic |
logstash-input-kafka |
|
log4j |
Reads events over a TCP socket from a Log4j SocketAppender object
|
logstash-input-log4j |
|
lumberjack |
Receives events using the Lumberjack protocl |
logstash-input-lumberjack |
|
rabbitmq |
Pulls events from a RabbitMQ exchange |
logstash-input-rabbitmq |
|
redis |
Reads events from a Redis instance |
logstash-input-redis |
|
s3 |
Streams events from files in a S3 bucket |
logstash-input-s3 |
|
sqs |
Pulls events from an Amazon Web Services Simple Queue Service queue |
logstash-input-sqs |
|
stdin |
Reads events from standard input |
logstash-input-stdin |
|
syslog |
Reads syslog messages as events |
logstash-input-syslog |
|
tcp |
Reads events from a TCP socket |
logstash-input-tcp |
|
|
Reads events from the Twitter Streaming API |
logstash-input-twitter |
|
udp |
Reads events over UDP |
logstash-input-udp |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
|
Plugin |
Description |
Github repository |
|
cloudwatch |
Pulls events from the Amazon Web Services CloudWatch API |
logstash-input-cloudwatch |
|
drupal_dblog |
Retrieves watchdog log events from Drupal installations with DBLog enabled |
logstash-input-drupal_dblog |
|
eventlog |
Pulls events from the Windows Event Log |
logstash-input-eventlog |
|
exec |
Captures the output of a shell command as an event |
logstash-input-exec |
|
ganglia |
Reads Ganglia packets over UDP |
logstash-input-ganglia |
|
gemfire |
Pushes events to a GemFire region |
logstash-input-gemfire |
|
github |
Reads events from a GitHub webhook |
logstash-input-github |
|
heroku |
Streams events from the logs of a Heroku app |
logstash-input-heroku |
|
imap |
Reads mail from an IMAP server |
logstash-input-imap |
|
irc |
Reads events from an IRC server |
logstash-input-irc |
|
jmx |
Retrieves metrics from remote Java applications over JMX |
logstash-input-jmx |
|
kinesis |
Receives events through an AWS Kinesis stream |
logstash-input-kinesis |
|
meetup |
Captures the output of command line tools as an event |
logstash-input-meetup |
|
pipe |
Streams events from a long-running command pipe |
logstash-input-pipe |
|
puppet_facter |
Receives facts from a Puppet server |
logstash-input-puppet_facter |
|
rackspace |
Receives events from a Rackspace Cloud Queue service |
logstash-input-rackspace |
|
relp |
Receives RELP events over a TCP socket |
logstash-input-relp |
|
rss |
Captures the output of command line tools as an event |
logstash-input-rss |
|
salesforce |
Creates events based on a Salesforce SOQL query |
logstash-input-salesforce |
|
snmptrap |
Creates events based on SNMP trap messages |
logstash-input-snmptrap |
|
sqlite |
Creates events based on rows in an SQLite database |
logstash-input-sqlite |
|
stomp |
Creates events received with the STOMP protocol |
logstash-input-stomp |
|
unix |
Reads events over a UNIX socket |
logstash-input-unix |
|
varnishlog |
Reads from the varnish cache shared memory log
|
logstash-input-varnishlog |
|
websocket |
Reads events from a websocket |
logstash-input-websocket |
|
wmi |
Creates events based on the results of a WMI query |
logstash-input-wmi |
|
xmpp |
Receives events over the XMPP/Jabber protocol |
logstash-input-xmpp |
|
zenoss |
Reads Zenoss events from the fanout exchange |
logstash-input-zenoss |
|
zeromq |
Reads events from a ZeroMQ SUB socket |
logstash-input-zeromq |
filter
|
Plugin |
Description |
Github repository |
|
aggregate |
Aggregates information from several events originating with a single task |
logstash-filter-aggregate |
|
anonymize |
Replaces field values with a consistent hash |
logstash-filter-anonymize |
|
csv |
Parses comma-separated value data into individual fields |
logstash-filter-csv |
|
date |
Parses dates from fields to use as the Logstash timestamp for an event |
logstash-filter-date |
|
de_dot |
Computationally expensive filter that removes dots from a field name |
logstash-filter-de_dot |
|
dissect |
Extracts unstructured event data into fields using delimiters |
logstash-filter-dissect |
|
dns |
Performs a standard or reverse DNS lookup |
logstash-filter-dns |
|
drop |
Drops all events |
logstash-filter-drop |
|
fingerprint |
Fingerprints fields by replacing values with a consistent hash |
logstash-filter-fingerprint |
|
geoip |
Adds geographical information about an IP address |
logstash-filter-geoip |
|
grok |
Parses unstructured event data into fields |
logstash-filter-grok |
|
json |
Parses JSON events |
logstash-filter-json |
|
kv |
Parses key-value pairs |
logstash-filter-kv |
|
multiline |
Merges multiple lines into a single event |
logstash-filter-multiline |
|
mutate |
Performs mutations on fields |
logstash-filter-mutate |
|
ruby |
Executes arbitrary Ruby code |
logstash-filter-ruby |
|
sleep |
Sleeps for a specified time span |
logstash-filter-sleep |
|
split |
Splits multi-line messages into distinct events |
logstash-filter-split |
|
syslog_pri |
Parses the PRI (priority) field of a syslog message
|
logstash-filter-syslog_pri |
|
throttle |
Throttles the number of events |
logstash-filter-throttle |
|
translate |
Replaces field contents based on a hash or YAML file |
logstash-filter-translate |
|
urldecode |
Decodes URL-encoded fields |
logstash-filter-urldecode |
|
useragent |
Parses user agent strings into fields |
logstash-filter-useragent |
|
uuid |
Adds a UUID to events |
logstash-filter-uuid |
|
xml |
Parses XML into fields |
logstash-filter-xml |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
|
Plugin |
Description |
Github repository |
|
alter |
Performs general alterations to fields that the mutate filter does not handle
|
logstash-filter-alter |
|
cidr |
Checks IP addresses against a list of network blocks |
logstash-filter-cidr |
|
cipher |
Applies or removes a cipher to an event |
logstash-filter-cipher |
|
clone |
Duplicates events |
logstash-filter-clone |
|
collate |
Collates events by time or count |
logstash-filter-collate |
|
elapsed |
Calculates the elapsed time between a pair of events |
logstash-filter-elapsed |
|
elasticsearch |
Copies fields from previous log events in Elasticsearch to current events |
logstash-filter-elasticsearch |
|
environment |
Stores environment variables as metadata sub-fields |
logstash-filter-environment |
|
extractnumbers |
Extracts numbers from a string |
logstash-filter-extractnumbers |
|
i18n |
Removes special characters from a field |
logstash-filter-i18n |
|
json_encode |
Serializes a field to JSON |
logstash-filter-json_encode |
|
metaevent |
Adds arbitrary fields to an event |
logstash-filter-metaevent |
|
metricize |
Takes complex events containing a number of metrics and splits these up into multiple events, each holding a single metric |
logstash-filter-metricize |
|
metrics |
Aggregates metrics |
logstash-filter-metrics |
|
oui |
Parse OUI data from MAC addresses |
logstash-filter-oui |
|
prune |
Prunes event data based on a list of fields to blacklist or whitelist |
logstash-filter-prune |
|
punct |
Strips all non-punctuation content from a field |
logstash-filter-punct |
|
range |
Checks that specified fields stay within given size or length limits |
logstash-filter-range |
|
tld |
Replaces the contents of the default message field with whatever you specify in the configuration |
logstash-filter-tld |
|
yaml |
Takes an existing field that contains YAML and expands it into an actual data structure within the Logstash event |
logstash-filter-yaml |
|
zeromq |
Sends an event to ZeroMQ |
logstash-filter-zeromq |
output
Elastic supported plugins
These plugins are maintained and supported by Elastic.
|
Plugin |
Description |
Github repository |
|
csv |
Writes events to disk in a delimited format |
logstash-output-csv |
|
elasticsearch |
Stores logs in Elasticsearch |
logstash-output-elasticsearch |
|
|
Sends email to a specified address when output is received |
logstash-output-email |
|
file |
Writes events to files on disk |
logstash-output-file |
|
graphite |
Writes metrics to Graphite |
logstash-output-graphite |
|
http |
Sends events to a generic HTTP or HTTPS endpoint |
logstash-output-http |
|
kafka |
Writes events to a Kafka topic |
logstash-output-kafka |
|
lumberjack |
Sends events using the lumberjack protocol
|
logstash-output-lumberjack |
|
rabbitmq |
Pushes events to a RabbitMQ exchange |
logstash-output-rabbitmq |
|
redis |
Sends events to a Redis queue using the RPUSHcommand
|
logstash-output-redis |
|
s3 |
Sends Logstash events to the Amazon Simple Storage Service |
logstash-output-s3 |
|
stdout |
Prints events to the standard output |
logstash-output-stdout |
|
tcp |
Writes events over a TCP socket |
logstash-output-tcp |
|
udp |
Sends events over UDP |
logstash-output-udp |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
|
Plugin |
Description |
Github repository |
|
boundary |
Sends annotations to Boundary based on Logstash events |
logstash-output-boundary |
|
circonus |
Sends annotations to Circonus based on Logstash events |
logstash-output-circonus |
|
cloudwatch |
Aggregates and sends metric data to AWS CloudWatch |
logstash-output-cloudwatch |
|
datadog |
Sends events to DataDogHQ based on Logstash events |
logstash-output-datadog |
|
datadog_metrics |
Sends metrics to DataDogHQ based on Logstash events |
logstash-output-datadog_metrics |
|
elasticsearch_java |
Stores logs in Elasticsearch using the node andtransport protocols
|
logstash-output-elasticsearch_java |
|
exec |
Runs a command for a matching event |
logstash-output-exec |
|
ganglia |
Writes metrics to Ganglia’s gmond
|
logstash-output-ganglia |
|
gelf |
Generates GELF formatted output for Graylog2 |
logstash-output-gelf |
|
google_bigquery |
Writes events to Google BigQuery |
logstash-output-google_bigquery |
|
google_cloud_storage |
Writes events to Google Cloud Storage |
logstash-output-google_cloud_storage |
|
graphtastic |
Sends metric data on Windows |
logstash-output-graphtastic |
|
hipchat |
Writes events to HipChat |
logstash-output-hipchat |
|
influxdb |
Writes metrics to InfluxDB |
logstash-output-influxdb |
|
irc |
Writes events to IRC |
logstash-output-irc |
|
jira |
Writes strutured JSON events to JIRA |
logstash-output-jira |
|
juggernaut |
Pushes messages to the Juggernaut websockets server |
logstash-output-juggernaut |
|
librato |
Sends metrics, annotations, and alerts to Librato based on Logstash events |
logstash-output-librato |
|
loggly |
Ships logs to Loggly |
logstash-output-loggly |
|
metriccatcher |
Writes metrics to MetricCatcher |
logstash-output-metriccatcher |
|
mongodb |
Writes events to MongoDB |
logstash-output-mongodb |
|
nagios |
Sends passive check results to Nagios |
logstash-output-nagios |
|
nagios_nsca |
Sends passive check results to Nagios using the NSCA protocol |
logstash-output-nagios_nsca |
|
newrelic |
Sends logstash events to New Relic Insights as custom events |
logstash-output-newrelic |
|
opentsdb |
Writes metrics to OpenTSDB |
logstash-output-opentsdb |
|
pagerduty |
Sends notifications based on preconfigured services and escalation policies |
logstash-output-pagerduty |
|
pipe |
Pipes events to another program’s standard input |
logstash-output-pipe |
|
rackspace |
Sends events to a Rackspace Cloud Queue service |
logstash-output-rackspace |
|
redmine |
Creates tickets using the Redmine API |
logstash-output-redmine |
|
riak |
Writes events to the Riak distributed key/value store |
logstash-output-riak |
|
riemann |
Sends metrics to Riemann |
logstash-output-riemann |
|
sns |
Sends events to Amazon’s Simple Notification Service |
logstash-output-sns |
|
solr_http |
Stores and indexes logs in Solr |
logstash-output-solr_http |
|
sqs |
Pushes events to an Amazon Web Services Simple Queue Serice queue |
logstash-output-sqs |
|
statsd |
Sends metrics using the statsd network daemon
|
logstash-output-statsd |
|
stomp |
Writes events using the STOMP protocol |
logstash-output-stomp |
|
syslog |
Sends events to a syslog server
|
logstash-output-syslog |
|
webhdfs |
Sends Logstash events to HDFS using the webhdfsREST API
|
logstash-output-webhdfs |
|
websocket |
Publishes messages to a websocket |
logstash-output-websocket |
|
xmpp |
Posts events over XMPP |
logstash-output-xmpp |
|
zabbix |
Sends events to a Zabbix server |
logstash-output-zabbix |
|
zeromq |
Writes events to a ZeroMQ PUB socket |
logstash-output-zeromq |
时间： 2024-11-05 19:08:03