部署kube-prometheus，添加邮件报警

这个项目出自coreos，已经存在很久了，第一次尝试的时候还很简陋，现在完善了很多。

项目提供了一键部署脚本，跑起来并不难，不过个人感觉要真正掌握并灵活使用并不是很容易。

kube version: 1.9.1

OS version: debian stretch

1、从github把项目拉下来。

# git clone https://github.com/coreos/prometheus-operator.git

2、准备镜像，这里用到了墙外的镜像，自行科学下载。

quay.io/prometheus/alertmanager:v0.9.1
quay.io/coreos/configmap-reload:v0.0.1
grafana/grafana:4.5.2
quay.io/coreos/grafana-watcher:v0.0.8
quay.io/coreos/kube-state-metrics:v1.0.1
gcr.io/google_containers/addon-resizer:1.0
quay.io/prometheus/node-exporter:v0.15.0
quay.io/prometheus/prometheus:v2.0.0
quay.io/coreos/prometheus-config-reloader:v0.0.2
quay.io/coreos/prometheus-operator:v0.15.0

3、执行脚本部署项目。

脚本内容如下：

#!/usr/bin/env bash

if [ -z "${KUBECONFIG}" ]; then
    export KUBECONFIG=~/.kube/config
fi

# CAUTION - setting NAMESPACE will deploy most components to the given namespace
# however some are hardcoded to ‘monitoring‘. Only use if you have reviewed all manifests.

if [ -z "${NAMESPACE}" ]; then
    NAMESPACE=monitoring
fi

kubectl create namespace "$NAMESPACE"

kctl() {
    kubectl --namespace "$NAMESPACE" "[email protected]"
}

kctl apply -f manifests/prometheus-operator

# Wait for CRDs to be ready.
printf "Waiting for Operator to register custom resource definitions..."
until kctl get customresourcedefinitions servicemonitors.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
until kctl get customresourcedefinitions prometheuses.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
until kctl get customresourcedefinitions alertmanagers.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
until kctl get servicemonitors.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
until kctl get prometheuses.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
until kctl get alertmanagers.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
echo "done!"

kctl apply -f manifests/node-exporter
kctl apply -f manifests/kube-state-metrics
kctl apply -f manifests/grafana/grafana-credentials.yaml
kctl apply -f manifests/grafana
find manifests/prometheus -type f ! -name prometheus-k8s-roles.yaml ! -name prometheus-k8s-role-bindings.yaml -exec kubectl --namespace "$NAMESPACE" apply -f {} \;
kubectl apply -f manifests/prometheus/prometheus-k8s-roles.yaml
kubectl apply -f manifests/prometheus/prometheus-k8s-role-bindings.yaml
kctl apply -f manifests/alertmanager/

从脚本上看，其实很简单，有木有..

# cd contrib/kube-prometheus/
# hack/cluster-monitoring/deploy
namespace "monitoring" created
clusterrolebinding "prometheus-operator" created
clusterrole "prometheus-operator" created
serviceaccount "prometheus-operator" created
service "prometheus-operator" created
deployment "prometheus-operator" created
Waiting for Operator to register custom resource definitions......done!
daemonset "node-exporter" created
service "node-exporter" created
clusterrolebinding "kube-state-metrics" created
clusterrole "kube-state-metrics" created
deployment "kube-state-metrics" created
rolebinding "kube-state-metrics" created
role "kube-state-metrics-resizer" created
serviceaccount "kube-state-metrics" created
service "kube-state-metrics" created
secret "grafana-credentials" created
secret "grafana-credentials" unchanged
configmap "grafana-dashboards-0" created
deployment "grafana" created
service "grafana" created
servicemonitor "alertmanager" created
servicemonitor "prometheus-operator" created
prometheus "k8s" created
servicemonitor "kubelet" created
servicemonitor "prometheus" created
service "prometheus-k8s" created
servicemonitor "node-exporter" created
servicemonitor "kube-scheduler" created
servicemonitor "kube-controller-manager" created
servicemonitor "kube-state-metrics" created
configmap "prometheus-k8s-rules" created
serviceaccount "prometheus-k8s" created
servicemonitor "kube-apiserver" created
role "prometheus-k8s" created
role "prometheus-k8s" created
role "prometheus-k8s" created
clusterrole "prometheus-k8s" created
rolebinding "prometheus-k8s" created
rolebinding "prometheus-k8s" created
rolebinding "prometheus-k8s" created
clusterrolebinding "prometheus-k8s" created
secret "alertmanager-main" created
service "alertmanager-main" created
alertmanager "main" created

4、由于事先准备好了镜像，很快就运行起来了。

# kubectl get po -n monitoring
NAME                                   READY     STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2       Running   0          11h
alertmanager-main-1                    2/2       Running   0          11h
alertmanager-main-2                    2/2       Running   0          11h
grafana-6b67b479d5-2hhnp               2/2       Running   0          11h
kube-state-metrics-6f7b5c94f-r8hm7     2/2       Running   0          11h
node-exporter-27744                    1/1       Running   0          11h
node-exporter-9vhlv                    1/1       Running   0          11h
node-exporter-rhjfb                    1/1       Running   0          11h
node-exporter-xpqr8                    1/1       Running   0          11h
prometheus-k8s-0                       2/2       Running   0          11h
prometheus-k8s-1                       2/2       Running   0          11h
prometheus-operator-8697c7fff9-mm8v5   1/1       Running   0          11h

这里曝光了三个服务：

  • Prometheus UI on node port 30900
  • Alertmanager UI on node port 30903
  • Grafana on node port 30902

通过相应地端口就能访问对应的服务。

5、添加controller-manager和scheduler的监控。

# kubectl apply -f -f manifests/k8s/ -n kube-system

这里其实是添加了2个service，注意namespace是kube-system，而不是monitoring：

# kubectl get ep -n kube-system | grep discovery
kube-controller-manager-prometheus-discovery   192.168.5.104:10252,192.168.5.105:10252,192.168.5.42:10252               3d
kube-scheduler-prometheus-discovery            192.168.5.104:10251,192.168.5.105:10251,192.168.5.42:10251               3d

6、由于我的kubernetes是高可用架构，有三个apiserver，这里有个bug有个需要处理。apiserver是无状态的，三个endpoint会自身冲突，需要是在kubernetes1.9以上，通过给apiserver传递一个参数--endpoint-reconciler-type=lease解决该问题。

默认是这个样子：

解决完是这个样子：

7、添加邮件报警规则。这个配置是用base64被加密的。

# vim manifests/alertmanager/alertmanager-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-main
data:
  alertmanager.yaml: Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KICBzbXRwX3NtYXJ0aG9zdDogIm1haWwub3VwZW5nLmNvbToyNSIKICBzbXRwX2Zyb206ICJuYWdpb3NfbW9uaXRvckBvdXBlbmcuY29tIgogIHNtdHBfYXV0aF91c2VybmFtZTogIm5hZ2lvc19tb25pdG9yQG91cGVuZy5jb20iCiAgc210cF9hdXRoX3Bhc3N3b3JkOiAiZGVsbGRlbGwiCnJvdXRlOgogIGdyb3VwX2J5OiBbJ2FsZXJ0bmFtZScsICdjbHVzdGVyJywgJ3NlcnZpY2UnXQogIGdyb3VwX3dhaXQ6IDMwcwogIGdyb3VwX2ludGVydmFsOiA1bQogIHJlcGVhdF9pbnRlcnZhbDogM2gKICByZWNlaXZlcjogdGVhbS1YLW1haWxzCiAgcm91dGVzOgogIC0gbWF0Y2hfcmU6CiAgICAgIGFsZXJ0bmFtZTogXihob3N0X2NwdV91c2FnZXxub2RlX2ZpbGVzeXN0ZW1fZnJlZXxob3N0X2Rvd24pJAogICAgcmVjZWl2ZXI6IHRlYW0tWC1tYWlscwogICAgcm91dGVzOgogICAgLSBtYXRjaDoKICAgICAgICBzZXZlcml0eTogY3JpdGljYWwKICAgICAgcmVjZWl2ZXI6IHRlYW0tWC1tYWlscwpyZWNlaXZlcnM6Ci0gbmFtZTogInRlYW0tWC1tYWlscyIKICBlbWFpbF9jb25maWdzOgogIC0gdG86ICJuaG9yaXpvbi1zYUBvdXBlbmcuY29tIgo=

要拿到配置内容，用base64反解一下就可以了，默认配置是这样：

# echo "加密的内容" | base64 -d
global:
  resolve_timeout: 5m
route:
  group_by: [‘job‘]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: ‘null‘
  routes:
  - match:
      alertname: DeadMansSwitch
    receiver: ‘null‘
receivers:
- name: ‘null‘

添加自己的邮件设置：

global:
  resolve_timeout: 5m
  smtp_smarthost: "mail.xxxx.com:25"
  smtp_from: "[email protected]"
  smtp_auth_username: "[email protected]"
  smtp_auth_password: "pass"
route:
  group_by: [‘alertname‘, ‘cluster‘, ‘service‘]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 15m
  receiver: team-X-mails
  routes:
  - match_re:
      alertname: ^(host_cpu_usage|node_filesystem_free|host_down)$
    receiver: team-X-mails
    routes:
    - match:
        severity: critical
      receiver: team-X-mails
receivers:
- name: "team-X-mails"
  email_configs:
  - to: "[email protected]"

定义好之后，用base64加密一下，替换掉之前的地方。然后应用配置：

# kubectl apply -f manifests/alertmanager/alertmanager-config.yaml -n monitoring

这时候就能收到报警了：

8、到这里部署就完成了，贴一下几个页面：

grafana

prometheus

alertmanager

原文地址：https://www.cnblogs.com/keithtt/p/8283795.html

时间： 2024-10-13 05:10:07

部署kube-prometheus，添加邮件报警的相关文章

prometheus altermanager邮件报警

prometheus altermanager邮件报警: 下载网址:https://prometheus.io/download tar zxfv alertmanager-0.15.2.linux-amd64.tar.gz -C /space/ mv /space/alertmanager-0.15.2.linux-amd64 /Influxdb/alertmanager vi /space/altermanager/altermanager.yml global:resolve_timeou

Zabbix部署、监测及邮件报警机制（实战！）

Zabbix部署 实验环境: CentOS 7-2:192.168.18.147(监测端:部署安装zabbix) CentOS 7-3:192.168.18.128(被监测端) 监测端操作: [[email protected] ~]# systemctl stop firewalld.service #关闭防火墙功能 [[email protected] ~]# systemctl disable firewalld.service #开机禁用防火墙功能 Removed symlink /et

zabbix告警（一）---添加邮件报警

说明:由于zabbix服务端和被监控端都已添加到监控系统中,服务也已正常运行.由于会发生宕机和服务终止等情况,现在需要添加告警,在Zabbix服务端设置邮件报警,当被监控主机宕机或者达到触发器预设值时,会自动发送报警邮件到指定邮箱. zabbix邮件报警分为两种情况: 1.Zabbix服务端只是单纯的发送报警邮件到指定邮箱,发送报警邮件的这个邮箱账号是Zabbix服务端的本地邮箱账号(例如:[email protected]),只能发送,不能接收外部邮件. 2.使用一个可以在互联网上正常收发邮件

分布式监控系统Zabbix3.2给异常添加邮件报警

在前一篇 分布式监控系统Zabbix3.2跳坑指南 中已安装好服务端和客户端,此处客户端是被监控的服务器,可能有上百台服务器.监控的目的一个是可以查看历史状态,可以对比零晨和工作区间数据的对比,以便后期进行优化指导.还有一个是报警,总不能等到服务器出现异常了才去从头查是什么问题吧.所以这篇主要介绍报警中最基础的一个 配置邮件预警. 通常zabbix提供了 e-mail.sms.jabber.微信等预警方式,sms等前期需要资金投入那就先否决吧,谁叫老板不给钱. 安装邮件发送工具mailx 这里我

Prometheus + AlertManager 邮件报警

安装 wget https://github.com/prometheus/alertmanager/releases/download/v0.13.0/alertmanager-0.13.0.linux-amd64.tar.gz tar -axvf alertmanager-0.13.0.linux-amd64.tar.gz 配置AlertManager AlertManager安装目录下有默认的simple.yml文件,可以创建新的配置文件,在启动时指定即可. 配置文件如下: global:

zabbix通过python脚本实现邮件报警

1.修改zabbix server端的配置文件,启用AlertScriptsPath,使其支持脚本功能 1.编辑zabbix server端的配置文件,并重启服务 # vim /etc/zabbix/zabbix_server.conf AlertScriptsPath=/usr/local/zabbix/alertscripts # /etc/init.d/zabbix_server restart 2.在服务端添加邮件报警的python脚本,并给脚本执行权限 邮件报警,并记录日志 # vim

分布式监控系统Zabbix-3.0.3-完整安装记录（5）-邮件报警部署

前面几篇陆续介绍了zabbix3.0.3监控系统的部署和监控项配置,今天这里分享下zabbix3.0.3的邮件报警的配置过程~ 由于采用sendmail发送邮件,常常会被认为是垃圾邮件被拒,所以不推荐这种方式! 这里,针对zabbix报警信息的发送,可以采用下面两种方式中的任意一种:(1)利用sendEmail程序来发送报警邮件.sendEmail是一个轻量级,命令行的SMTP邮件客户端,非常完美,使用简单并且功能强大.这个被设计用在php.bash .perl和web站点使用.(2)利用pyt

在LAMP架构中部署zabbix监控系统及邮件报警机制

初步了解zabbix: Zabbix能监视各种网络参数,保证服务器系统的安全运营:并提供灵活的通知机制以让系统管理员快速定位/解决存在的各种问题.是一个基于WEB界面的提供分布式系统监视以及网络监视功能的企业级的开源解决方案.被监控对象只要支持SNMP协议或者运行zabbix_agents代理程序即可.agent端:主机通过安装agent方式采集数据.server端:通过收集agent发送的数据,写入数据库(MySQL,ORACLE等),再通过php+apache在web前端展示. SNMP:

Centos7下zabbix部署（四）定义报警媒介-邮件

1.安装发送邮件工具mailx [[email protected] ~]# yum install mailx -y 2.自定义使用163邮箱为默认发件人(避免被当作垃圾邮件) set from=13311802282@163.com set smtp=smtp.163.com set smtp-auth-user=username set smtp-auth-password=a7260488 set smtp-auth=login 注意:163邮箱需要开启一个授权密码,在非163专用客户端