最近做日志分析，发现logstash较符合自己的需求，
- Logstash：做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助.一般使用shipper作为log收集、indexer作为log转载.
- Logstash shipper收集log 并将log转发给redis 存储
- Logstash indexer从redis中读取数据并转发给elasticsearch
- redis：是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。
- Elasticsearch：elasticsearch是基于lucene的开源搜索引擎,用来做索引。
- Kibana： 开源web展现,界面很漂亮,是一个功能强大的elasticsearch数据显示客户端，logstash已经内置了kibana，你也可以单独部署kibana，最新版的kibana3是纯html+js客户端.
软件下载目录
http://www.elasticsearch.org/downloads/
我的环境如下
os：centos6.3_x86-64
redis-2.8.7.tar.gz
kibana-3.0.0
java version "1.7.0_51"
elasticsearch-0.90.12
一，安装java
yum -y install java
二,安装redis
cd ~/src wget http://download.redis.io/releases/redis-2.8.7.tar.gz tar -zxf redis-2.8.7.tar.gz cd redis-2.8.7.tar.gz make sudo make install
安装完毕后
/etc/init.d/redis_6379 start
测试是否正常
[[email protected] ~]# redis-cli ping PONG [[email protected] ~]#
[[email protected] ~]# netstat -tanpu|grep redis tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 1391/redis-server *
三，安装Elasticsearch
cd /search sudo mkdir elasticsearch cd elasticsearch sudo wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.12.zip sudo unzip elasticsearch-0.90.12.zip
备注：当开始使用的是1.x.x java报错，后来用的0.9.。
https://groups.google.com/forum/#!topic/logstash-users/fvFT7pgQTEM
Are you using elasticsearch_http for your output? If not, 1.3.3 is based on 0.90.x elasticsearch, and won’t play nice with 1.0.x elasticsearch with just the “elasticsearch” output.
启动ES服务器
切换到elasticsearch目录运行
bin/elasticsearch -f
默认端口是9200
curl -X GET http://localhost:9200
[[email protected] ~]# curl -X GET http://localhost:9200
{
"ok" : true,
"status" : 200,
"name" : "Master Pandemonium",
"version" : {
"number" : "0.90.12",
"build_hash" : "26feed79983063ae83bfa11bd4ce214b1f45c884",
"build_timestamp" : "2014-02-25T15:38:23Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
},
"tagline" : "You Know, for Search"
}
四.安装logstash
cd /search sudo mkdir logstash cd logstash sudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar
新建配置文件index.conf
# This is the logstash server index configuration.
# This file will be put in the same folder with logtash.jar file in the
# /etc/logtash/
# This takes information straight from redis and loads it into elasticsearch.
input {
redis {
host => "127.0.0.1"
type => "syslog"
threads => 4
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use json_event here since the sender is a logstash agent
format => "json_event"
}
}
output {
elasticsearch {
host => "127.0.0.1"
}
}
新建shiper.conf
input {
stdin {
type => "test"
}
}
output {
stdout { codec => rubydebug }
redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}
运行配置
java -jar logstash.jar agent -f shipper.conf java -jar logstash.jar agent -f index.conf
五，配置kibana
logstash的最新版已经内置kibana，你也可以单独部署kibana。kibana3是纯粹JavaScript+html的客户端，所以可以部署到任意http服务器上。
https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.zip
解压到web目录
http://127.0.0.1/kibana/index.html
资料来源参考：
1.http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html
3.http://my.oschina.net/guol/blog/179848
4.http://tinytub.github.io/logstash-install.html
5.install in Ubuntu server
http://tips4admin.com/blog/2013/10/how-to-centralize-your-log-with-logstash-elasticsearch-redis-kibana-in-ubuntu-server/
6.logstash官方文档 the logstash book
Kibana+Logstash+Elasticsearch+Redis安装部署