ELK日志收集部署

1. 1.   时间同步：

ntpdate pool.ntp.org

echo ‘*/5 * * * * ntpdate pool.ntp.org’>> /var/spool/cron/root

1. 2.   关闭防火墙和selinux

/etc/init.d/iptables  stop

chkconfig iptables off

Sed –I ‘s/SELINUX=enforcing/SELINUX=disabled/g’  /etc/selinux/config

1. 3.   安装java

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

tar zxf jdk-7u45-linux-x64.tar.gz

mv jdk1.8.0_92/ /usr/local/jdk

设置jdk环境变量

vi /etc/profile

-------------------------------------------------------

JAVA_HOME=/usr/local/jdk

PATH=$PATH:$JAVA_HOME/bin

CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

export JAVA_HOME PATHCLASSPATH

-------------------------------------------------------

source /etc/profile

java -version

1. 4.   下载redis

wget http://download.redis.io/releases/redis-3.2.3.tar.gz
tar zxf redis-3.2.3.tar.gz
cd redis-3.2.3
make 
make PREFIX=/usr/local/redis install
mkdir /usr/local/redis/conf
cp  redis.conf /usr/local/redis/conf/redis.conf.bak
cd /usr/local/redis/conf
cp redis.conf.bak redis.conf

添加环境变量
echo ‘PATH=$PATH:/usr/local/redis/bin/‘>>/etc/profile
source /etc/profile

启动redis：

/usr/local/redis/bin/redis-server/usr/local/redis/conf/redis.conf &

1. 5.   下载logstash  elasticsearch  kibana

https://www.elastic.co/downloads

elasticsearch-5.0.0.tar.gz

logstash-5.0.0.tar.gz

kibana-5.0.0-linux-x86_64.tar.gz

1. 6.   解压文件：

tar zxf logstash-5.0.0.tar.gz

tar zxf elasticsearch-5.0.0.tar.gz

tar zxf kibana-5.0.0-linux-x86_64.tar.gz

1. 7.   移动到统一管理目录：

mv elasticsearch-5.0.0 /usr/local/elasticsearch

mv logstash-5.0.0 /usr/local/logstash

mv kibana-5.0.0-linux-x86_64 /usr/local/kibana

1. 8.   备份配置文件：

cp /usr/local/logstash/config/logstash.yml /usr/local/logstash/config/logstash.yml.bak.$(date+%F)

cp /usr/local/elasticsearch/config/elasticsearch.yml/usr/local/elasticsearch/config/elasticsearch.yml.bak.$(date +%F)

cp /usr/local/kibana/config/kibana.yml/usr/local/kibana/config/kibana.yml.bak.$(date +%F)

1. 9.  配置Elasticsearch

创建用户

默认elasticsearch不支持root用户启动，所以需要先创建一个普通用户

groupadd elastic

useradd elastic –g  elastic –M

chown -R elastic.elastic  /usr/local/elasticsearch/

修改配置文件：

network.host: 192.168.0.248

http.port: 9200

su elastic

/usr/local/elasticsearch/bin/elasticsearch –d

验证启动：

curl http://localhost:9200/

添加开机启动：

echo ‘/usr/local/elasticsearch/bin/elasticsearch-d’ >>/etc/rc.local

注意错误：

ERROR: bootstrap checks failed

问题：max file descriptors [65535] for elasticsearchprocess likely too low, increase to at least [65536]

解决：vi/etc/security/limits.conf

*             -       nofile          65536

或者

* soft nofile 65536

* hard nofile 131072

* soft nproc 2048

* hard nproc 4096

问题：max number of threads [1024] for user[elasticsearch] likely too low, increase to at least [2048]

解决：vi/etc/security/limits.d/90-nproc.conf

* soft nproc 2048

需重启生效。

问题：max virtual memory areas vm.max_map_count [65530]likely too low, increase to at least [262144]

解决：vi/etc/sysctl.conf

vm.max_map_count=655360

sysctl -p

1. 10.       配置kibana：

修改配置文件：

vi /usr/local/kibana/config/kibana.yml

server.port: 5601

server.host: "192.168.0.248"

elasticsearch.url: http://192.168.0.248:9200

1. 11.       配置logstash

测试logstash：

cd logstash-5.0.0

bin/logstash -e ‘input { stdin { } } output {stdout {} }‘

hello world

2013-11-21T01:22:14.405+0000 0.0.0.0 helloworld

配置logstash服务端：

vi /usr/local/logstash/config/logstash.yml

input {

redis {

host => “192.168.0.248”

port =>6379

type =>”redis-input”

data_type=>”list”

key =>”logstash:redis”

}

}

output {

elasticsearch{

hosts => ["192.168.0.248:9200"]

}

}

客户端：

vi /usr/local/logstash/config/logstash.yml

input {

file{

type=>”nginx_access”

path=>[”/usr/local/nginx/logs/access.log”]

}

}

output {

redis{

host  => “192.168.0.248”

data_type=> “list”

key=> “logstash:redis”

}

}

启动客户端：

/usr/local/logstash/bin/logstash –f /usr/local/logstash/conf/logstash.conf

或yum安装：

/usr/share/logstash/bin/logstash -f/etc/logstash/conf.d/logstash.conf

1. 12.1
2. 13.1

客户端yum安装：

rpm --importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch

cat > /etc/yum.repos.d/logstash.repo <<EOF

[logstash-5.x]

name=Elastic repository for 5.x packages

baseurl=https://artifacts.elastic.co/packages/5.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

EOF

yum clean all

yum install logstash –y