亿级PV的ELK集群实践之路

前言

笔者多年前便维护过ELK，但是由于站点日志流量及服务器数量并不是很多基本都是单机搞定。

然而光Web服务器就400+，Nginx日志大小每天50G+，加上其他业务系统日志，之前单机ELK显然不足以支撑现有的业务场景。

规划篇

目前的业务采用阿里云+自建机房的模式，阿里云做为线上业务，自建机房做为灾备中心，尽可能的将线上日志实时传输到自建机房进行数据分析。

架构图

简述

1.日志集中处理

笔者一开始是在每台机器通过filebeat+logstash的方式将日志进行收集和处理后发送到elasticsearch，logstash本身java应用比较耗费内存，而且维护成本较高。

后来采用rsyslog的方式将所有服务器存储到单台阿里云服务器，再通过rsyslog转发到自建机房，基本实现了毫秒级的同步。

2.解耦

起初是通过Logstash直接往Elasticsearch存储日志，一旦遇到需要重启或者维护Elasticsearch集群的时候这时日志将无处安放，难免造成日志丢失.

引入Redis后即便elasticsearch在维护期间也可以先将数据缓存下来。将Logstash-shipper和Logstash-indexer划清了界限。

3.集群多实例

自建机房预备了3台内存为256G服务器部署ELK集群，但是官方建议jvm的内存不要超过32G，大概原因是一旦jvm设置超过32G将会采用不同的算法，这种算法会耗费更多系统资源。

比如设置为48G的情况下性能甚至不及20G，具体解释参考官方链接。

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

所以为了不浪费资源我在每台机器上部署了两个Elasticsearch节点，共6个node.

4.收集日志类型

  • nginx-access.log
  • nginx-error.log
  • php-error.log
  • php-slow.log
  • action.log 前面几个都好理解，这个日志是开发采集用户后台管理操作关键行为的json格式日志。

5.服务器规划

主机名和公网IP均做了化名处理.ELK stak3台物理机跑了6个Elasticsearch集群node，6个logstash-indexer，1个logstash-shipper，1个redis，1个kibana，1个nginx.

主机名 配置 用途 备注
rsyslog-relay 8core 16G 1T 收集所有服务器所需日志，集中存储转发。 系统为centos7.3
elk01 16core 256G 9.8T
nginx

kibana

rsyslog-server

logstash-shipper

elk01-indexer

elk01-indexer2

elk01-elasticsearch

elk01-elasticsearch2
3台服务器配置均相同

系统均为centos7.3

磁盘为12块1.8T 10k的sas盘，做Raid 0+1.
elk02 16core 256G 9.8T
elk02-indexer

elk02-indexer2

elk02-elasticsearch

elk02-elasticsearch2
elk03 16core 256G 9.8T
elk03-indexer

elk03-indexer2

elk03-elasticsearch

elk03-elasticsearch2

部署篇

rsyslog

升级

将日志集中存储到本地机房,Centos7.3自带的rsyslog为V7版本，先升级到V8。因为V8的rsyslog-relp有日志重传机制，可防止数据丢失。

卸载原有版本，添加v8 yum源，安装新版本。

[[email protected] ~]# rpm -qa|grep rsyslog
rsyslog-7.4.7-16.el7.x86_64[[email protected] ~]# yum remove -y rsyslog-7.4.7-16.el7.x86_64vim /etc/yum.repos.d/rsyslog_v8.repo
[rsyslog_v8]
baseurl = http://rpms.adiscon.com/v8-stable/epel-$releasever/$basearch
enabled = 1
gpgcheck = 1
gpgkey = http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon
name = Rsyslog version 8 repository

[[email protected] ~]#yum install rsyslog rsyslog-relp -y

rsyslog客户端配置

nginx的文件名根据业务区分为后台管理访问日志，前台访问日志，支付日志，但是根据类型打上了nginx-access和nginx-error两个标签。

配置文件路径为/etc/rsyslog.conf

 1 $ModLoad imuxsock
 2 $ModLoad imjournal
 3 $ModLoad imfile
 4 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
 5 $IncludeConfig /etc/rsyslog.d/*.conf
 6 *.info;mail.none;authpriv.none;cron.none                /var/log/messages
 7 authpriv.*                                              /var/log/secure
 8 mail.*                                                  -/var/log/maillog
 9 cron.*                                                  /var/log/cron
10 *.emerg                                                 :omusrmsg:*
11 uucp,news.crit                                          /var/log/spooler
12 local7.*                                                /var/log/boot.log
13
14 ##########Start Nginx Log File#################
15 $InputFileName /var/log/nginx/access-admin.log
16 $InputFileTag site-web1-nginx-access:
17 $InputFileStateFile site-web1-nginx-access
18 $InputFileSeverity debug
19 $InputRunFileMonitor
20 $InputFilePollInterval 1
21
22 $InputFileName /var/log/nginx/error-admin.log
23 $InputFileTag site-web1-nginx-error:
24 $InputFileStateFile site-web1-nginx-error
25 $InputFileSeverity debug
26 $InputRunFileMonitor
27 $InputFilePollInterval 1
28
29 $InputFileName /var/log/nginx/access-frontend.log
30 $InputFileTag site-web1-nginx-access:
31 $InputFileStateFile site-web1-nginx-access
32 $InputFileSeverity debug
33 $InputRunFileMonitor
34 $InputFilePollInterval 1
35
36 $InputFileName /var/log/nginx/error-frontend.log
37 $InputFileTag site-web1-nginx-error:
38 $InputFileStateFile site-web1-nginx-error
39 $InputFileSeverity debug
40 $InputRunFileMonitor
41 $InputFilePollInterval 1
42
43 $InputFileName /var/log/nginx/access-pay.log
44 $InputFileTag site-web1-nginx-access:
45 $InputFileStateFile site-web1-nginx-access
46 $InputFileSeverity debug
47 $InputRunFileMonitor
48 $InputFilePollInterval 1
49
50 $InputFileName /var/log/nginx/error-pay.log
51 $InputFileTag site-web1-nginx-error:
52 $InputFileStateFile site-web1-nginx-error
53 $InputFileSeverity debug
54 $InputRunFileMonitor
55 $InputFilePollInterval 1
56 ######################End Of Nginx Log File################
57
58 ######################Start Of Action Log File#############
59 $InputFileName /var/log/php-fpm/action_log.log
60 $InputFileTag site-web1-action:
61 $InputFileStateFile site-web1-action
62 $InputFileSeverity debug
63 $InputRunFileMonitor
64 $InputFilePollInterval 1
65 ######################End Of Action Log File###############
66
67 #####################Start PHP Log File###################
68 $InputFileName /var/log/php-fpm/www-slow.log
69 $InputFileTag site-web1-php-slow:
70 $InputFileStateFile site-web1-php-slow
71 $InputFileSeverity debug
72 $InputRunFileMonitor
73 $InputFilePollInterval 1
74 $InputFileReadMode 2
75
76 $InputFileName /var/log/php-fpm/error.log
77 $InputFileTag site-web1-php-error:
78 $InputFileStateFile site-web1-php-error
79 $InputFileSeverity debug
80 $InputRunFileMonitor
81 $InputFilePollInterval 1
82
83 $WorkDirectory /var/lib/rsyslog
84 $ActionQueueType LinkedList
85 $ActionQueueFileName srvrfwd
86 $ActionResumeRetryCount -1
87 $ActionQueueSaveOnShutdown on
88 ####################End Of PHP log File#####################
89
90 ###################Start Log Forward##############################################
91 if $programname == ‘site-web1-nginx-access‘ then @@阿里云rsyslog服务器内网地址:514
92 if $programname == ‘site-web1-nginx-error‘ then @@阿里云rsyslog服务器内网地址:514
93 if $programname == ‘site-web1-php-slow‘ then @@阿里云rsyslog服务器内网地址:514
94 if $programname == ‘site-web1-php-error‘ then @@阿里云rsyslog服务器内网地址:514
95 if $programname == ‘site-web1-action‘ then @@阿里云rsyslog服务器内网地址:514
96 ###################End Of log Forward##############################################

rsyslog阿里云中继服务器配置(rsyslog-relay)

1.主配置文件路径为/etc/rsyslogconf，每台服务器对应的配置文件通过include的方式放置在

/etc/rsyslog.d/，文件名以.conf结尾，对应主配置文件的第7行。
 1 $ModLoad omrelp
 2 $ModLoad imudp
 3 $UDPServerRun 514
 4 $ModLoad imtcp
 5 $InputTCPServerRun 514
 6 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
 7 $IncludeConfig /etc/rsyslog.d/*.conf
 8 $umask 0022
 9 *.info;mail.none;authpriv.none;cron.none                /var/log/messages
10 authpriv.*                                              /var/log/secure
11 mail.*                                                  -/var/log/maillog
12 cron.*                                                  /var/log/cron
13 *.emerg                                                 :omusrmsg:*
14 uucp,news.crit                                          /var/log/spooler
15 local7.*                                                /var/log/boot.log
16 *.* :omrelp:本地机房rsyslog服务器:20514

2.每台服务器配置文件，如site-web1的示例如下：

 1 $template site-web1-nginx-access,"/data/rsyslog/nginx/site/site-web1-nginx-access.log"
 2 $template site-web1-nginx-error,"/data/rsyslog/nginx/site/site-web1-nginx-error.log"
 3 $template site-web1-php-slow,"/data/rsyslog/php/site/site-web1-php-slow.log"
 4 $template site-web1-php-error,"/data/rsyslog/php/site/site-web1-php-error.log"
 5 $template site-web1-action,"/data/rsyslog/php/site/site-web1-action.log"
 6
 7 if $programname == ‘site-web1-nginx-access‘ then ?site-web1-nginx-access
 8 if $programname == ‘site-web1-nginx-error‘ then ?site-web1-nginx-error
 9 if $programname == ‘site-web1-php-slow‘ then ?site-web1-php-slow
10 if $programname == ‘site-web1-php-error‘ then ?site-web1-php-error
11 if $programname == ‘site-web1-action‘ then ?site-web1-action

本地机房rsyslog配置(elk01)

/etc/rsyslog.conf主配置文件如下，另外/etc/rsyslog.d/里面的站点配置文件跟中继服务器里面的一模一样。

 1 $ModLoad imrelp
 2 $ModLoad omrelp
 3 $InputRELPServerRun 20514
 4 $WorkDirectory /var/lib/rsyslog
 5 $DirCreateMode 0755
 6 $FileCreateMode 0644
 7 $FileOwner logstash
 8 $DirOwner logstash
 9 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
10 $IncludeConfig /etc/rsyslog.d/*.conf
11 $OmitLocalLogging on
12 $IMJournalStateFile imjournal.state
13 *.info;mail.none;authpriv.none;cron.none                /var/log/messages
14 authpriv.*                                              /var/log/secure
15 mail.*                                                  -/var/log/maillog
16 cron.*                                                  /var/log/cron
17 *.emerg                                                 :omusrmsg:*
18 uucp,news.crit                                          /var/log/spooler
19 local7.*                                                /var/log/boot.log
20 $PrivDropToGroup logstash

至此所有服务器日志都通过rsyslog集中收集了.

elk安装

均是通过yum方式安装的最新6.x版本,根据规划elk01上安装的nginx及elk02上安装的redis均是采用yum方式安装，就不在一一赘述。

elasticsearch yum源

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

logstash yum源

[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

kibana yum源

[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

logstash-shipper配置

logstash安装后配置文件路径默认为/etc/logstash，拷贝一份做为logstash-shipper的配置文件目录.

cp -r /etc/logstash /etc/logstash-shipperchown -R logstash.logstash /etc/logstash-shipper

主配置文件/etc/logstash-shipper/logstash.yml

path.data: /var/lib/logstash-shipper
path.config: /etc/logstash-shipper/conf.d
path.logs: /var/log/logstash/shipper

创建相关目录并授权予logstash用户

mkdir -p /var/lib/logstash-shipper && chown logstash.logstash /var/lib/logstash-shippermkdir -p /var/log/logstash/shipper && chown logstash.logstash /var/log/logstash/shipper

站点配置文件/etc/logstash-shipper/conf.d/shipper.conf

截至目前站点配置有3千多行，全部贴出来略显冗余，这里挑一个站的配置供参考。

input {
    file {
         path => "/data/rsyslog/php/site/site-*-php-error.log"
         type => "site-php-error"
         sincedb_path => "/data/sincedb/site"
         }

    file {
         path => "/data/rsyslog/php/site/site-*-php-slow.log"
         type => "site-php-slow"
         sincedb_path => "/data/sincedb/site"
         }

    file {
         path => "/data/rsyslog/nginx/site/site-*-nginx-error.log"
         type => "site-nginx-error"
         sincedb_path => "/data/sincedb/site"
         }

    file {
         path => "/data/rsyslog/nginx/site/site-*-nginx-access.log"
         type => "site-nginx-access"
         sincedb_path => "/data/sincedb/site"
         }

    file {
         path => "/data/rsyslog/php/site/site*action.log"
         type => "site-action"
         sincedb_path => "/data/sincedb/site"
         }
}

output {
       redis {
         host => "elk02内网地址"
         port => "6379"
         db => "8"
         data_type => "list"
         key => "server-log"
    }
}

服务器启动脚本:/etc/systemd/system/logstash-shipper.service

[Unit]
Description=logstash-shipper

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with ‘-‘ makes it try to load, but if the file doesn‘t
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash-shipper"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

logstash-indexer配置

每台服务器均跑了两个indexer

logstash-indexer1

拷贝配置文件目录

cp -r /etc/logstash /etc/logstash-indexerchown -R logstash.logstash /etc/logstash-indexer

主配置文件/etc/logstash-indexer/logstash.yml

path.data: /var/lib/logstash
path.config: /etc/logstash-indexer/conf.d
path.logs: /var/log/logstash/indexer

创建相关目录并授权予logstash用户

mkdir -p /var/log/logstash/indexer && chown logstash.logstash /var/log/logstash/indexer

每个日志类型都对应了一个配置文件，放置在/etc/logstash-indexer/conf.d

/etc/logstash-indexer/conf.d
├── action.conf            程序自定义的用户行为日志
├── nginx_access.conf      nginx访问日志
├── nginx_error.conf       nginx错误日志
├── php_error.conf         php错误日志
└── php_slow.conf          php-slow日志

各配置文件如下

action.conf

input {
    redis {
        host => "elk02内网ip"
        port => "6379"
        db => "8"
        data_type => "list"
        key => "kosun-log"
    }
}

filter {
    if [type] =~ ‘^.+action‘ {
      mutate {
      gsub => [ "message", "^.+-action: ", "" ]
             }
     json {
       source => "message"
           }
      date {
          match => ["time", "yyyy-MM-dd HH:mm:ss"]
          target => "@timestamp"
          "locale" => "en"
          timezone => "Asia/Shanghai"
          remove_field => ["time"]
            }
     }
}

output {
    if [type] =~ ‘^.+action‘ {
        elasticsearch {
            hosts => ["elasticsearch:9200"]
            index => "logstash-action-%{+YYY.MM.dd}"
        }
    }
}

nginx-access

input {
    redis {
        host => "elk02内网地址"
        port => "6379"
        db => "8"
        data_type => "list"
        key => "kosun-log"
    }
}

filter {
    if [type] =~ ‘^.+nginx-access‘ {
        fingerprint {
            method => "SHA1"
            key    => "^.+nginx-access"
        }
        grok {
            match => [ "message" , "%{COMBINEDAPACHELOG} %{DATA:msec} %{QUOTEDSTRING:x_forward_ip} %{DATA:server_name} %{DATA:request_time} %{DATA:upstream_response_time} %{DATA:scheme} %{GREEDYDATA:extra_fields}" ]
            overwrite => [ "message" ]
        }

        date {
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss.SSS Z"]
            #target => "@timestamp"
            "locale" => "en"
            timezone => "Asia/Shanghai"
        }

        mutate {
            gsub => ["agent", "\"", ""]
            gsub => ["referrer", "\"", ""]
            gsub => ["x_forward_ip", "\"", ""]
            gsub => ["extra_fields", "\"", ""]
        }

        if [extra_fields] =~ /^{.*}$/ {
            mutate {
                gsub => ["extra_fields", "\"","",
                    "extra_fields", "\\x0A","",
                    "extra_fields", "\\x22",‘\"‘,
                    "extra_fields", "(\\)",""
                    ]
            }

            json {
                source => "extra_fields"
                target => "extra_fields_json"
            }
        }

        geoip {
            source => "clientip"
            fields => ["city_name","location"]
        }
    }
}

output {
    if [type] =~ ‘^.+nginx-access‘ {
        elasticsearch {
            hosts => ["elasticsearch:9200"]
            index => "logstash-%{type}-%{+YYYY.MM.dd}"
            document_type => "%{type}"
           # flush_size => 50000
           # idle_flush_time => 10
            sniffing => true
            template_overwrite => true
            document_id => "%{fingerprint}"
        }
    }
}

nginx-error

input {
       redis {
         host => "elk02内网地址"
         port => "6379"
         db => "8"
         data_type => "list"
         key => "kosun-log"
       }
}

filter {
   if [type] =~ ‘^.+nginx-error‘ {
     fingerprint {
       method => "SHA1"
         key    => "^.+nginx-error"
        }
  grok {
    match => {
        "message" => [
            "(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[%{DATA:err_severity}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:client_ip})?(?:, referrer: \"%{URI:referrer})?",

            "%{DATESTAMP:timestamp} \[%{DATA:err_severity}\] %{GREEDYDATA:err_message}"
        ]
      }
    }
    date {
        match => ["timestamp" , "YYYY/MM/dd HH:mm:ss"]
        "locale" => "en"
        timezone => "Asia/Shanghai"
        remove_field => [ "timestamp" ]
        }
     }
}

output {
      if [type] =~ ‘^.+nginx-error‘ {
        elasticsearch {
           hosts => ["elasticsearch:9200"]
           index => "logstash-nginx-error-%{+YYY.MM.dd}"
           document_id => "%{fingerprint}"
     }
   }
}

php-error

input {
       redis {
         host => "elk02内网地址"
         port => "6379"
         db => "8"
         data_type => "list"
         key => "kosun-log"
       }
}

output {
    if [type] =~ ‘^.+php-error‘ {
       elasticsearch {
       hosts => ["elasticsearch:9200"]
       index => "logstash-php-error-%{+YYY.MM.dd}"
     }
  }
}

php-slow

input {
       redis {
         host => "elk02内网地址"
         port => "6379"
         db => "8"
         data_type => "list"
         key => "kosun-log"
       }
}

filter {
    if [type] =~ ‘^.+slow‘ {
        multiline {
            pattern => "\[\d{2}-"
            negate => true
            what => "previous"
        }
    }
}

output {
     if [type] =~ ‘^.+slow‘ {
       elasticsearch {
       hosts => ["elasticsearch:9200"]
       index => "logstash-php-slow-%{+YYY.MM.dd}"
     }
  }
}

服务启动脚本/etc/systemd/system/logstash-indexer.service

[Unit]
Description=logstash-indexer

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with ‘-‘ makes it try to load, but if the file doesn‘t
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash-indexer"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

　　logstash-indexer2的配置文件一样，只需修改相应的目录和启动脚本即可。

Elasticsearch集群部署

每台服务器跑了两个elasticsearch实例，一个为yum安装，一个为源码包解压。

yum默认安装的配置文件位于/etc/elasticsearch

以elk01为例elasticsearch实例1配置

修改jvm.options里面的内存设置为31g

-Xms31g
-Xmx31g

主配置文件/etc/elasticsearch/elasticsearch.yml

#============================cluster setting==============================
cluster.name:elk
cluster.routing.allocation.same_shard.host: true
#============================node setting=================================
node.name: elk01
node.master: true
node.data: true
#============================path setting=================================
path.data: /data/es-data
path.logs: /var/log/elasticsearch
#============================memory setting===============================
bootstrap.memory_lock: false
#============================network setting==============================
network.host: elk01内网地址
http.port: 9200
transport.tcp.port: 9300
#============================thread_pool setting==========================
thread_pool.search.queue_size: 10000
#============================discovery setting============================
discovery.zen.ping.unicast.hosts: ["elk01:9300", "elk02:9300", "elk03:9300"]
discovery.zen.minimum_master_nodes: 2
#============================gateway setting==============================
gateway.recover_after_nodes: 4
gateway.recover_after_time: 5m
gateway.expected_nodes: 5
indices.recovery.max_bytes_per_sec: 800mb
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false

创建数据目录并授权

mkdir -p /data/es-data && chown elasticsearch.elasticsearch /data/es-data

服务启动脚本,yum安装自带，无需修改/usr/lib/systemd/system/elasticsearch.service

elk01 elasticsearch实例2配置

该实例是用源码包做的

下载源码包，解压，移动到指定目录，授权

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.2.ziptar xf elasticsearch-6.1.2.zipmv elasticsearch-6.1.2 /usr/local/chown -R elasticsearch.elasticsearch /usr/local/elasticsearch-6.1.2

修改/usr/local/elasticsearch-6.1.2/config/jvm.options 内存设置31g

配置文件/usr/local/elasticsearch-6.1.2/config/elasticsearch.yml

#============================cluster setting==============================
cluster.name: elk
cluster.routing.allocation.same_shard.host: true
#node.max_local_storage_nodes: 2
#============================node setting=================================
node.name: elk01-2
node.master: false
node.data: true
#============================path setting=================================
path.data: /data/es-data2
path.logs: /var/log/elasticsearch2
#============================memory setting===============================
bootstrap.memory_lock: false
#============================network setting==============================
network.host: elk01内网ip
http.port: 9201
transport.tcp.port: 9301
#============================thread_pool setting==========================
thread_pool.search.queue_size: 10000
#============================discovery setting============================
discovery.zen.ping.unicast.hosts: ["elk01:9300", "elk02:9300", "elk03:9300"]
discovery.zen.minimum_master_nodes: 2
#============================gateway setting==============================
gateway.recover_after_nodes: 4
gateway.recover_after_time: 5m
gateway.expected_nodes: 5
indices.recovery.max_bytes_per_sec: 800mb
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false

创建相关目录并授权

mkdir -p /data/es-data2 && chown elasticsearch.elasticsearch /data/es-data2

mkdir -p /var/log/elasticsearch2 elasticsearch.elasticsearch /var/log/elasticsearch2

服务启动脚本/etc/init.d/elasticsearch2

#!/bin/bash
#
# elasticsearch <summary>
#
# chkconfig:   2345 80 20
# description: Starts and stops a single elasticsearch instance on this system
#

### BEGIN INIT INFO
# Provides: Elasticsearch
# Required-Start: $network $named
# Required-Stop: $network $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: This service manages the elasticsearch daemon
# Description: Elasticsearch is a very scalable, schema-free and high-performance search solution supporting multi-tenancy and near realtime search.
### END INIT INFO

#
# init.d / servicectl compatibility (openSUSE)
#
if [ -f /etc/rc.status ]; then
    . /etc/rc.status
    rc_reset
fi

#
# Source function library.
#
if [ -f /etc/rc.d/init.d/functions ]; then
    . /etc/rc.d/init.d/functions
fi

# Sets the default values for elasticsearch variables used in this script
ES_HOME="/usr/share/elasticsearch2"
MAX_OPEN_FILES=65536
MAX_MAP_COUNT=262144
ES_PATH_CONF="/etc/elasticsearch2"

PID_DIR="/var/run/elasticsearch2"

# Source the default env file
ES_ENV_FILE="/etc/sysconfig/elasticsearch"
if [ -f "$ES_ENV_FILE" ]; then
    . "$ES_ENV_FILE"
fi

exec="$ES_HOME/bin/elasticsearch"
prog="elasticsearch"
pidfile="$PID_DIR/${prog}.pid"

export ES_JAVA_OPTS
export JAVA_HOME
export ES_PATH_CONF
export ES_STARTUP_SLEEP_TIME

lockfile=/var/lock/subsys/$prog

start() {

    # Ensure that the PID_DIR exists (it is cleaned at OS startup time)
    if [ -n "$PID_DIR" ] && [ ! -e "$PID_DIR" ]; then
        mkdir -p "$PID_DIR" && chown elasticsearch:elasticsearch "$PID_DIR"
    fi
    if [ -n "$pidfile" ] && [ ! -e "$pidfile" ]; then
        touch "$pidfile" && chown elasticsearch:elasticsearch "$pidfile"
    fi

    cd $ES_HOME
    echo -n $"Starting $prog: "
    # if not running, start it up here, usually something like "daemon $exec"
    #daemon --user elasticsearch --pidfile $pidfile $exec -p $pidfile -d
    su - elasticsearch -c "${exec} -p ${pidfile} -d"

    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    # stop it here, often "killproc $prog"
    killproc -p $pidfile -d 86400 $prog
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    stop
    start
}

reload() {
    restart
}

force_reload() {
    restart
}

rh_status() {
    # run checks to determine if the service is running or use generic status
    status -p $pidfile $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?

Kibana

elasticsearch均添加了hosts文件解析到本机内网IP

配置文件/etc/kibana/kibana.yml

egrep -v ‘^$|#‘ /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://elasticsearch:9200"
elasticsearch.requestTimeout: 3000000

Nginx server配置

server {
    listen 80;
    server_name 域名;

    access_log /var/log/nginx/access-elk.log;
    error_log /var/log/nginx/error-elk.log;

    location ~ /.well-known/acme-challenge {
        allow all;

        content_by_lua_block {
            auto_ssl:challenge_server()
        }
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name 域名;

    ssl_certificate_by_lua_block {
        auto_ssl:ssl_certificate()
    }

    ssl_certificate /etc/nginx/ssl/elk.crt;
    ssl_certificate_key /etc/nginx/ssl/elk.key;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:16m;
    ssl_dhparam /etc/nginx/ssl/dhparam.pem;

    access_log /var/log/nginx/acess-elk.log;
    error_log /var/log/nginx/error-elk.log warn;

    allow 开放访问IP段;
    deny all;

    location / {
        proxy_pass http://elasticsearch:5601;
        proxy_set_header Host $host;
    }
}

到这里配置部署篇基本已经完成，后续篇幅将介绍日常运维中常见的操作。

比如一键部署脚本，配置增删，批量部署，index，shard，常见操作，性能监控，优化，扩容......

先上个现在集群运行1个星期左右的x-pack监控截图，共采集2,448,834,111条记录，3T数据，日均约3亿条。

结语:

这篇博文断断续续大概花了将近1个星期整理完，本来想详细到每行配置文件都加上注解，熟话说万事开头难，好歹在博客园也开了个头了。

此时网易云音乐随机到一首乔杉唱的《塑料袋》颇有共鸣，突然觉得他是个被喜剧耽误了的胖歌手。

原文地址：https://www.cnblogs.com/david-qing/p/8527811.html

时间： 2024-10-02 02:32:59

亿级PV的ELK集群实践之路的相关文章

亿级PV超大型网站集群架构图形深度揭秘讲解

猛戳下面地址观看: 亿级PV超大型网站集群架构图形深度揭秘讲解

从100PV到1亿级PV网站架构演变

如果你对项目管理.系统架构有兴趣,请加微信订阅号"softjg",加入这个PM.架构师的大家庭 一个网站就像一个人,存在一个从小到大的过程.养一个网站和养一个人一样,不同时期需要不同的方法,不同的方法下有共同的原则.本文结合我自已14年网站人的经历记录一些架构演变中的体会. 1:积累是必不可少的 架构师不是一天练成的. 1999年,我作了一个个人主页,在学校内的虚拟空间,参加了一次主页大赛,几个DREAMWEAVER的页面,几个TABLE作布局,一个DB连接,几行PHP的代码嵌入在HT

从100PV到1亿级PV站点架构演变

假设你对项目管理.系统架构有兴趣,请加微信订阅号"softjg".增加这个PM.架构师的大家庭 一个站点就像一个人,存在一个从小到大的过程. 养一个站点和养一个人一样.不同一时候期须要不同的方法,不同的方法下有共同的原则. 本文结合我自已14年站点人的经历记录一些架构演变中的体会. 1:积累是不可缺少的 架构师不是一天练成的. 1999年,我作了一个个人主页,在学校内的虚拟空间,參加了一次主页大赛,几个DREAMWEAVER的页面.几个TABLE作布局,一个DB连接,几行PHP的代码嵌

从100PV到1亿级PV网站架构演变（转）

http://www.linuxde.net/2013/05/13581.html 一个网站就像一个人,存在一个从小到大的过程.养一个网站和养一个人一样,不同时期需要不同的方法,不同的方法下有共同的原则.本文结合我自已14年网站人的经历记录一些架构演变中的体会. 积累是必不可少的 架构师不是一天练成的. 1999年,我作了一个个人主页,在学校内的虚拟空间,参加了一次主页大赛,几个DREAMWEAVER的页面,几个TABLE作布局,一个DB连接,几行PHP的代码嵌入在HTML中,再用ftp传到服务

转：亿级Web系统的高容错性实践（好博文）

亿级Web系统的高容错性实践 亿级Web系统的高容错性实践 背景介绍 大概三年前,我在腾讯负责的活动运营系统,因为业务流量规模的数倍增长,系统出现了各种各样的异常,当时,作为开发的我,7*24小时地没日没夜处理告警,周末和凌晨也经常上线,疲于奔命.后来,当时的老领导对我说:你不能总扮演一个“救火队长”的角色, 要尝试从系统整体层面思考产生问题的根本原因,然后推进解决. 我幡然醒悟,“火”是永远救不完的,让系统能够自动”灭火”,才是解决问题的正确方向.简而言之,系统的异常不能总是依赖于“人”去恢复

腾讯大规模Hadoop集群实践

腾讯大规模Hadoop集群实践 转自:http://www.csdn.net/article/2014-02-19/2818473-Tencent-Hadoop ID lishilong404740787 TDW是腾讯最大的离线数据处理平台.本文主要从需求.挑战.方案和未来计划等方面,介绍了TDW在建设单个大规模集群中采取的JobTracker分散化和NameNode高可用两个优化方案. TDW(Tencent distributed Data Warehouse,腾讯分布式数据仓库)基于开源软

大规模Hadoop集群实践：腾讯分布式数据仓库（TDW）

TDW 是腾讯最大的离线数据处理平台.本文主要从需求.挑战.方案和未来计划等方面,介绍了TDW在建设单个大规模集群中采取的 JobTracker 分散化和 NameNode 高可用两个优化方案. TDW(Tencent distributed Data Warehouse,腾讯分布式数据仓库)基于开源软件 Hadoop 和 Hive 进行构建,打破了传统数据仓库不能线性扩展.可控性差的局限,并且根据腾讯数据量大.计算复杂等特定情况进行了大量优化和改造. TDW服务覆盖了腾讯绝大部分业务产品,单集

配置三台服务器组成的ELK集群（二）

上一篇里主要是介绍了ES和ES-Head的安装过程,这一篇继续介绍ELK集群的其他核心组件安装过程. 五.安装Logstash: 本案的Logstash安装在10.113.130.117上:燃鹅,Logstash也可以利用多台组成集群,如果未来单台处理不过来,可以把Logstash扩展到其他服务器上. 将logstash.tar.gz解压到指定目录: # tar -zxvf logstash-5.4.1.tar.gz -C /opt/ # mv /opt/logstash-5.4.1/ /opt

通过docker搭建ELK集群

单机ELK,另外两台服务器分别有一个elasticsearch节点,这样形成一个3节点的ES集群. 可以先尝试单独搭建es集群或单机ELK https://www.cnblogs.com/lz0925/p/12018209.html // 单机ELK https://www.cnblogs.com/lz0925/p/12011026.html // 三节点es集群 注意点 服务器内存:要求不低于8G,如果4G,没有跑其他程序的话,应该也可以,低于4G基本不用考虑. 我的系统:阿里云centOS7