最近做日志分析，发现logstash较符合自己的需求，
Logstash:做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助。一般使用shipper作为log收集、indexer作为log转载。
Logstash shipper收集log 并将log转发给redis 存储
Logstash indexer从redis中读取数据并转发给elasticsearch
redis:是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。
Elasticsearch:elasticsearch是基于lucene的开源搜索引擎,用来做索引。
Kibana: 开源web展现,界面很漂亮,是一个功能强大的elasticsearch数据显示客户端，logstash已经内置了kibana，你也可以单独部署kibana，最新版的kibana3是纯html+js客户端。
软件下载目录
http://www.elasticsearch.org/downloads/
我的环境如下
IP:192.168.81.44 t44 OS:CentOS6.5 x86_64 openjdk version "1.8.0_31" nginx-1.0.15 redis-2.4.10 elasticsearch-1.5.0 logstash-1.4.2 kibana-3.1.0
一、配置epel YUM源
yum -y localinstall http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
二、安装jdk环境
yum -y install java-1.8.0-openjdk
三、安装redis、启动redis
yum -y install redis ; /etc/init.d/redis restart
四、安装配置Elasticsearch、启动Elasticsearch
wget -c https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.0.tar.gz -O /root/elasticsearch-1.5.0.tar.gz tar -xvf /root/elasticsearch-1.5.0.tar.gz -C /usr/local/
添加2行配置:
tail -n2 /usr/local/elasticsearch-1.5.0/config/elasticsearch.yml http.cors.allow-origin: "/.*/" http.cors.enabled: true
启动Elasticsearch
/usr/local/elasticsearch-1.5.0/bin/elasticsearch -d
查看Elasticsearch日志
tail -f /usr/local/elasticsearch-1.5.0/logs/elasticsearch.log
五、安装配置Logstash、启动Logstash
wget -N https://download.elastic.co/logstash/logstash/logstash-1.4.2.tar.gz -O /root/logstash-1.4.2.tar.gz tar -xvf /root/logstash-1.4.2.tar.gz -C /usr/local/
配置index.conf
cat /usr/local/logstash-1.4.2/bin/index.conf
input {
redis {
host => "127.0.0.1"
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use the ‘json‘ codec here because we expect to read
# json events from redis.
codec =>json
}
file {
type =>"t44message"
path =>["/var/log/messages"]
}
syslog {
type => "rsyslog"
port => 514
}
file {
type =>"t44secure"
path =>["/var/log/secure"]
}
file {
type =>"t44nginx"
path =>["/var/log/nginx/*.log"]
}
}
output {
# stdout { debug => true debug_format => "json"}
stdout { codec =>rubydebug }
elasticsearch {
host => "127.0.0.1"
}
}
启动Logstash
/usr/local/logstash-1.4.2/bin/logstash -f /usr/local/logstash-1.4.2/bin/index.conf
六、安装nginx、kibana
yum -y install nginx wget -c https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz -O /root/kibana-3.1.0.tar.gz tar -xvf /root/kibana-3.1.0.tar.gz -C /usr/share/nginx/html/
配置nginx:
egrep -v ‘^#| #‘ /etc/nginx/conf.d/default.conf |grep -v ‘^$‘
server {
listen 80 default_server;
server_name _;
include /etc/nginx/default.d/*.conf;
location / {
root /usr/share/nginx/html;
index index.html index.htm index.php;
}
error_page 404 /404.html;
location = /404.html {
root /usr/share/nginx/html;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location ~ \.php$ {
root /usr/share/nginx/html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_buffer_size 32k;
fastcgi_buffers 8 32k;
include fastcgi_params;
}
}
配置kibana:
grep ‘elasticsearch:‘ /usr/share/nginx/html/kibana/config.js |grep -v ‘\*‘
elasticsearch: "http://192.168.81.44:9200",
\cp /usr/share/nginx/html/kibana/app/dashboards/default.json{,.bak}
\cp /usr/share/nginx/html/kibana/app/dashboards/logstash.json /usr/share/nginx/html/kibana/app/dashboards/default.json
访问kibana:
http://192.168.81.44/kibana/
时间： 2024-10-12 03:49:12