系统版本:centos 6.4 x86.64
网络环境:我这台机器处于内网,连不了外网,所以要通过代理上
服务端:elasticsearch和kibana、Fluentd Aggregator安装在一台机器上 IP为192.168.1.5
客户端:192.168.1.6 Mysql Server
192.168.1.7:tomcat Server
架构图:
注意:1、2、3步服务端和客户端都要安装
1、首先安装ruby
首先把系统自带的ruby删除啦,系统自带ruby版本太低
安装依赖软件
yum -y install zlib-devel libcurl-devel
接着安装yaml
tar xvf /soft/yaml-0.1.4.tar.gz cd yaml-0.1.4 ./configure&&make&&make install
继续安装ruby
tar xvf /soft/ruby-1.9.3-p484.tar.gz cd ruby-1.9.3-p484 ./configure --prefix=/usr/local/ruby --enable-shared --disable-install-doc --with-opt-dir=/usr/local/ && make &&make install
2、安装td-agent
下载 http://174.129.37.216/redhat/x86_64/td-agent-1.1.17-0.x86_64.rpm
下载 http://174.129.37.216/redhat/x86_64/td-libyaml-0.1.4-1.x86_64.rpm
软件包放在/soft下
yum -y install /soft/td-*
3、安装td-agent插件
删除自带的ruby源,国外的源太慢了
/usr/lib64/fluent/ruby/bin/fluent-gem sources --remove http://rubygems.org/
添加淘宝的源
/usr/lib64/fluent/ruby/bin/fluent-gem sources --http-proxy http://172.17.17.132:235 -a http://ruby.taobao.org/
安装secure-forward插件
/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-secure-forward
4、安装elasticsearch(在服务端安装,需JDK1.7)
去官网下载RPM包
rpm -ivh /soft/elasticsearch-1.3.2.noarch.rpm
启动
/etc/init.d/elasticsearch start
5、安装kibana(在服务端安装)
去官网下载压缩包,直接放在网站目录下(nginx和apache都可以),纯静态页面写的
6、配置Fluentd Aggregator (在服务端安装)
默认打开24284端口,确保防火墙已开放此端口
安装fluent-plugin-elasticsearch插件,
/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-elasticsearch
编辑td-agent配置文件:/etc/td-agent/td-agent.conf
# Listen to incoming data over SSL <source> type secure_forward shared_key 123456 #密码设置,随便设,保持两边一致就行 self_hostname logserver.test.com cert_auto_generate yes </source> # Store Data in Elasticsearch <match *.**> type copy <store> type elasticsearch host localhost port 9200 include_tag_key true tag_key log_name logstash_format true flush_interval 1s </store> </match>
7、抓取mysql慢查询日志(客户端 192.168.1.6)
安装fluent-plugin-mysqlslowquery插件
/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-mysqlslowquery
编辑td-agent配置文件:/etc/td-agent/td-agent.conf
<source> type mysql_slow_query path /var/log/mysql/slow.log tag master.slow_query </source> <match *.**> type secure_forward shared_key 123456 #密码设置 self_hostname master.test.com <server> host 192.168.1.5 </server> </match>
8、抓取tomcat日志(客户端 192.168.1.7)
我两个tomcat实例,所以配了两个source
编辑td-agent配置文件:/etc/td-agent/td-agent.conf
<source> type tail path /usr/local/tomcat/logs/catalina.out pos_file /var/log/td-agent/tomcat_2_8080.log.pos tag tomcat_2_8080 #format /^(?<error_time>(\[.+?\]|^\t\S.*))(?<flag>\s.+?\s)(?<info>([\s\S]*))|(?<msg>([\s\S]*))$/ format /^(?<message>([\s\S]*))$/ </source> <source> type tail path /usr/local/tomcat1/logs/catalina.out pos_file /var/log/td-agent/tomcat_2_8081.log.pos tag tomcat_2_8081 format /^(?<message>([\s\S]*))$/ #format /^(?<error_time>(\[.+?\]|^\t\S.*))(?<flag>\s.+?\s)(?<info>([\s\S]*))|(?<msg>([\s\S]*))$/ </source> <match *.**> type secure_forward shared_key 123456 self_hostname tomcat.test.com <server> host 192.168.1.5 </server> </match>