55 logstach应用详解、ELK Stack

01 logstash应用详解

配置环境：

node3 192.168.1.133 CentOS Linux release 7.2

node4 192.168.1.134 CentOS Linux release 7.2

[[email protected] ~]# cd /etc/logstash/conf.d/

[[email protected] conf.d]# vim filesample.conf

input {

file {

path    =>  ["/var/log/messages"]

type    =>  "system"

start_position  =>  "beginning"

}   

}   

output {

stdout {

codec   =>  rubydebug

}   

}   

[[email protected] conf.d]# logstash -f filesample.conf --configtest

Configuration OK

[[email protected] ~]# rpm -ivh epel-release-latest-7.noarch.rpm 

[[email protected] ~]# yum -y install collectd  

[[email protected] ~]# vim /etc/collectd.conf

修改

#Hostname    "localhost"

为

Hostname    "node3"

修改

#LoadPlugin df

为

LoadPlugin df #监控磁盘

修改

#LoadPlugin network

为

LoadPlugin network

在<Plugin netlink>程序端后添加

<Plugin network>

    <Server "192.168.1.134" "25826">

    </Server>

</Plugin>

[[email protected] ~]# systemctl start collectd.service 

[[email protected] conf.d]# vim udpsample.conf

input {

udp {

port    =>  25826

codec   =>  collectd {}

type    =>  "collectd"

}   

}   

output {

stdout {

codec   =>  rubydebug

}   

} 

[[email protected] conf.d]# logstash -f udpsample.conf --configtest

Configuration OK

[[email protected] conf.d]# logstash -f udpsample.conf

[[email protected] conf.d]# yum -y install httpd

[[email protected] conf.d]# systemctl start httpd.service

[[email protected] conf.d]# vim groksample.conf

input {

stdin {}

}

filter {

grok {

match   =>  { "message" =>  "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }

}

}

output {

stdout {

codec   =>  rubydebug

}

}

[[email protected] conf.d]# logstash -f groksample.conf --configtest

Configuration OK

[[email protected] conf.d]# logstash -f groksample.conf

Logstash startup completed

1.1.1.1 GET /index.html 30 0.23

{

       "message" => "1.1.1.1 GET /index.html 30 0.23",

      "@version" => "1",

    "@timestamp" => "2017-01-03T13:37:24.978Z",

          "host" => "node4",

      "clientip" => "1.1.1.1",

        "method" => "GET",

       "request" => "/index.html",

         "bytes" => "30",

      "duration" => "0.23"

}

[[email protected] conf.d]# vim apachelogsample.conf

input {

    file {

        path            =>  ["/var/log/httpd/access_log"]

        type            =>  "apachelog"

        start_position  =>  "beginning"

    }

}

filter {

    grok {

        match       =>  { "message" =>  "%{COMBINEDAPACHELOG}" }

    }

}

output {

    stdout {

        codec       =>  rubydebug

    }

}

[[email protected] conf.d]# logstash -f apachelogsample.conf --configtest

Configuration OK   

[[email protected] conf.d]# vim /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns

在末尾添加

# nginx Logs

NGUSERNAME [a-zA-Z\.\@\-\+_%]+

NGUSER %{NGUSERNAME}

NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{OS:agent} %{NOTSPACE:http_x_forwarded_for}  

[[email protected] conf.d]# systemctl stop httpd.service

[[email protected] conf.d]# yum -y install nginx

[[email protected] conf.d]# systemctl start nginx.service

[[email protected] conf.d]# cd /var/log/nginx/

[[email protected] nginx]# ls

access.log  error.log

[[email protected] nginx]# tail access.log 

192.168.1.204 - - [03/Jan/2017:22:18:03 +0800] "GET / HTTP/1.1" 200 3700 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" "-"

192.168.1.204 - - [03/Jan/2017:22:18:03 +0800] "GET /nginx-logo.png HTTP/1.1" 200 368 "http://192.168.1.134/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" "-"

192.168.1.204 - - [03/Jan/2017:22:18:03 +0800] "GET /poweredby.png HTTP/1.1" 200 2811 "http://192.168.1.134/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" "-"

[[email protected] conf.d]# cd -

[[email protected] conf.d]# cp apachelogsample.conf nginxlogsample.conf  

[[email protected] conf.d]# vim nginxlogsample.conf 

input {

    file {

        path            =>  ["/var/log/nginx/access.log"]

        type            =>  "nginxlog"

        start_position  =>  "beginning"

    }

}

filter {

    grok {

        match       =>  { "message" =>  "%{NGINXACCESS}" }

    }

}

output {

    stdout {

        codec       =>  rubydebug

    }

}

[[email protected] conf.d]# logstash -f nginxlogsample.conf 

02 ELK Stack

[[email protected] ~]# yum install redis

[[email protected] ~]# vim /etc/redis.conf 

修改

bind 127.0.0.1

为

bind 0.0.0.0

[[email protected] ~]# systemctl start redis.service

[[email protected] ~]# redis-cli

127.0.0.1:6379> help

redis-cli 2.8.19

Type: "help @<group>" to get a list of commands in <group>

      "help <command>" for help on <command>

      "help <tab>" to get a list of possible help topics

      "quit" to exit

[[email protected] ~]# cd /etc/logstash/conf.d/

[[email protected] conf.d]# cp nginxlogsample.conf nglogredissample.conf

[[email protected] conf.d]# vim nglogredissample.conf 

input {

    file {

        path            =>  ["/var/log/nginx/access.log"]

        type            =>  "nginxlog"

        start_position  =>  "beginning"

    }

}

filter {

    grok {

        match       =>  { "message" =>  "%{NGINXACCESS}" }

    }

}

output {

    redis {

        port        =>  6379

        host        =>  ["127.0.0.1"]

        data_type   =>  "list"

key         =>  "logstash-%[type]"

    }

}

[[email protected] conf.d]# logstash -f nglogredissample.conf --configtest

Configuration OK

 

[[email protected] ~]# vim /etc/profile.d/java.sh

export JAVA_HOME=/usr

[[email protected] ~]# yum install -y logstash-1.5.4-1.noarch.rpm 

[[email protected] ~]# cd /etc/logstash/conf.d/

[[email protected] conf.d]# vim server.conf

input {

    redis {

        port        =>  "6370"

        host        => "192.168.1.134"

        data_type   =>  "list"

        key         =>  "logstash-nginxlog"

    }   

}   

output {

    stdout {

        codec       =>  rubydebug

    }   

}   

[[email protected] conf.d]# vim /etc/profile.d/logstash.sh

export PATH=/opt/logstash/bin:$PATH

[[email protected] conf.d]# . /etc/profile.d/logstash.sh

[[email protected] conf.d]# logstash -f server.conf --configtest

Configuration OK

[[email protected] ~]# yum makecache

[[email protected] ~]# yum install java-1.7.0-openjdk-devel.x86_64 

[[email protected] ~]# vim /etc/profile.d/java.sh

export JAVA_HOME=/usr

[[email protected] ~]# yum install elasticsearch-1.7.2.noarch.rpm -y

[[email protected] ~]# vim /etc/elasticsearch/elasticsearch.yml

修改

#cluster.name: elasticsearch

为

cluster.name: loges

修改

#node.name: "Franz Kafka"

为

node.name: "node1"

[[email protected] ~]# systemctl daemon-reload

[[email protected] ~]# systemctl start elasticsearch.service

[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i bigedsk -u file:///root/bigdesk-latest.zip 

[[email protected] ~]# tar xf kibana-4.1.2-linux-x64.tar.gz -C /usr/local/

[[email protected] ~]# cd /usr/local/

[[email protected] local]# ln -s kibana-4.1.2-linux-x64/ kibana

[[email protected] local]# cd kibana

[[email protected] kibana]# ls

bin  config  LICENSE.txt  node  plugins  README.txt  src

[[email protected] kibana]# cd config/

[[email protected] config]# vim kibana.yml 

修改

elasticsearch_url: "http://localhost:9200"

为

elasticsearch_url: "http://192.168.1.131:9200"

#启动  

[[email protected] kibana]# /usr/local/kibana/bin/kibana 

[[email protected] conf.d]# vim server.conf 

input {

    redis {

        port        =>  "6370"

        host        => "192.168.1.134"

        data_type   =>  "list"

        key         =>  "logstash-nginxlog"

    }

}

output {

    elasticsearch {

        cluster     =>  "loges"

        index       =>  "logstash-%{+YYYY.MM.dd}"

    }

}

[[email protected] conf.d]# logstash -f server.conf --configtest

Configuration OK

[[email protected] conf.d]# logstash -f server.conf

[[email protected] ~]# curl -XGET ‘localhost:9200/_cat/indices‘

yellow open .kibana 1 1 1 0 2.4kb 2.4kb 

该节视频到71:55(65382)由于错误太多无法继续进行

时间： 2024-10-11 13:26:37

55 logstach应用详解、ELK Stack的相关文章

Java基础(55):Exception类详解（转）

Java中的异常 Exception java.lang.Exception类是Java中所有异常的直接或间接父类.即Exception类是所有异常的根类. 比如程序: 1 public class ExceptionTest 2 { 3 public static void main(String[] args) 4 { 5 int a = 3; 6 int b = 0; 7 int c = a / b; 8 System.out.println(c); 9 } 10 } 编译通过,执行时结果

Logstach配置文件详解

配置文件的基本格式,输入部分,过滤器部分和输出部分. # This is a comment. You should use comments to describe # parts of your configuration. input {   ... } filter {   ... } output {   ... } 每个部分都可以配置一个或多个插件.下面展示的是输入部分,有两个file插件. input {   file {     path => "/var/log/mess

ELK+Filebeat 集中式日志解决方案详解

原文:ELK+Filebeat 集中式日志解决方案详解 链接:https://www.ibm.com/developerworks/cn/opensource/os-cn-elk-filebeat/index.html?ca=drs- ELK Stack 简介 ELK 不是一款软件,而是 Elasticsearch.Logstash 和 Kibana 三种软件产品的首字母缩写.这三者都是开源软件,通常配合使用,而且又先后归于 Elastic.co 公司名下,所以被简称为 ELK Stack.根据

ELK技术栈之-Logstash详解

ELK技术栈之-Logstash详解 前言 在第九章节中,我们已经安装好Logstash组件了,并且启动实例测试它的数据输入和输出,但是用的是最简单的控制台标准输入和标准输出,那这节我们就来深入的学习Logstash的详细使用. 常用启动参数 我们在上一节中演示了启动Logstash的实例,其中我们启动的时候给Logstash脚本传入了-e的参数,但实际上,Logstash的启动参数有很多,我们来看一下各个启动参数的作用: -e #立即启动实例,例如:./logstash -e "input {

Linux01-Linux日志系统syslog详解55

一.日志系统 1.日志系统:syslog() 信息详细程序:日志级别 日志子系统:facility 日志记录动作:Action Linux上的日志系统: syslog syslog-ng: syslog-ng作为syslog的替代工具,可以完全替代syslog的服务,并且通过定义规则,实现更好的过滤功能 系统启动过程的日志: kernel --> 物理终端(/dev/console) --> /var/log/dmesg 查看日志命令: dmesg /var/log/dmesg cat /va

Scala 深入浅出实战经典 第55讲：Scala中Infix Type实战详解

王家林亲授<DT大数据梦工厂>大数据实战视频 Scala 深入浅出实战经典(1-64讲)完整视频.PPT.代码下载: 百度云盘:http://pan.baidu.com/s/1c0noOt6 腾讯微云:http://url.cn/TnGbdC 360云盘:http://yunpan.cn/cQ4c2UALDjSKy 访问密码 45e2土豆:http://www.tudou.com/programs/view/9JKSqMiQuBE/优酷:http://v.youku.com/v_show/id

Android task和back stack详解(官方文档翻译)

一个应用往往包含很多activities．每个activity都应围绕着用户可执行的特定动作来设计,并且可以启动其它activitie．例如,一个email应用可能可能有一个显示新邮件列表的activity．当用户选择一个邮件,一个新的activity被打开以显示邮件内容． 一个activity也可以打开同一设备上存在于其它应用的activitie,例如,如果你的应用想要发送一个邮件,你可以定义一个intent来执行一个"send"动作并包含一些数据,比如一个地址和一条信息．另一个应用

Logstash安装（图文详解）（多节点的ELK集群安装在一个节点就好）

前提 Elasticsearch-2.4.3的下载(图文详解) Elasticsearch-2.4.3的单节点安装(多种方式图文详解) Elasticsearch-2.4.3的3节点安装(多种方式图文详解) 继续

iOS开发——UI篇OC篇&amp;UIStackView详解

UIStackView详解 一.继承关系.遵守协议.隶属框架及可用平台 UIStackView 类提供了一个高效的接口用于平铺一行或一列的视图组合.Stack视图使你依靠自动布局的能力,创建用户接口使得可以动态的调整设备朝向.屏幕尺寸及任何可用范围内的变化.Stack视图管理着所有在它的 arrangedSubviews 属性中的视图的布局.这些视图根据它们在 arrangedSubviews 数组中的顺序沿着 Stack 视图的轴向排列.精确的布局变量根据 Stack 视图的 axis , d