kubernetes集群全栈监控报警方案kube-prometheus

参考文档 

http://www.servicemesher.com/blog/prometheus-operator-manual/
https://github.com/coreos/prometheus-operator
https://github.com/coreos/kube-prometheus

背景环境

  • kubernetes集群1.13版本，纯二进制版本打造，参考k8s1.13集群部署

  • coreos/kube-prometheus从coreos/prometheus-operator独立出来了，后续entire monitoring stack只能去coreos/kube-prometheus
  • 目前在该环境下部署还没有遇到坑

监控原理

Prometheus读取Metrcs，读取etcd或者api的都行

查看etcd的metrics输出信息

[[email protected] yaml]# curl --cacert /k8s/etcd/ssl/ca.pem --cert /k8s/etcd/ssl/server.pem --key /k8s/etcd/ssl/server-key.pem https://10.2.8.34:2379/metrics

查看kube-apiserver的metrics信息

[[email protected] yaml]#  kubectl get --raw /metrics

实施部署

注意：This will be the last release supporting Kubernetes 1.13 and before. The next release is going to support Kubernetes 1.14+ only.
后续版本只支持k8s1.14+，所以后续要下载release版本，目前只有一个版本所以可以直接git clone

1.下载原代码

[[email protected] yaml]# git clone https://github.com/coreos/kube-prometheus
Cloning into ‘kube-prometheus‘...
remote: Enumerating objects: 5803, done.
remote: Total 5803 (delta 0), reused 0 (delta 0), pack-reused 5803
Receiving objects: 100% (5803/5803), 3.69 MiB | 536.00 KiB/s, done.
Resolving deltas: 100% (3441/3441), done.

2.查看原配置文件

[[email protected] yaml]# cd kube-prometheus/manifests
[[email protected] manifests]# ls
00namespace-namespace.yaml
0prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
0prometheus-operator-0prometheusCustomResourceDefinition.yaml
0prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
0prometheus-operator-0servicemonitorCustomResourceDefinition.yaml
0prometheus-operator-clusterRoleBinding.yaml
0prometheus-operator-clusterRole.yaml
0prometheus-operator-deployment.yaml
0prometheus-operator-serviceAccount.yaml
0prometheus-operator-serviceMonitor.yaml
0prometheus-operator-service.yaml
alertmanager-alertmanager.yaml
alertmanager-secret.yaml
alertmanager-serviceAccount.yaml
alertmanager-serviceMonitor.yaml
alertmanager-service.yaml
grafana-dashboardDatasources.yaml
grafana-dashboardDefinitions.yaml
grafana-dashboardSources.yaml
grafana-deployment.yaml
grafana-serviceAccount.yaml
grafana-serviceMonitor.yaml
grafana-service.yaml
kube-state-metrics-clusterRoleBinding.yaml
kube-state-metrics-clusterRole.yaml
kube-state-metrics-deployment.yaml
kube-state-metrics-roleBinding.yaml
kube-state-metrics-role.yaml
kube-state-metrics-serviceAccount.yaml
kube-state-metrics-serviceMonitor.yaml
kube-state-metrics-service.yaml
node-exporter-clusterRoleBinding.yaml
node-exporter-clusterRole.yaml
node-exporter-daemonset.yaml
node-exporter-serviceAccount.yaml
node-exporter-serviceMonitor.yaml
node-exporter-service.yaml
prometheus-adapter-apiService.yaml
prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
prometheus-adapter-clusterRoleBindingDelegator.yaml
prometheus-adapter-clusterRoleBinding.yaml
prometheus-adapter-clusterRoleServerResources.yaml
prometheus-adapter-clusterRole.yaml
prometheus-adapter-configMap.yaml
prometheus-adapter-deployment.yaml
prometheus-adapter-roleBindingAuthReader.yaml
prometheus-adapter-serviceAccount.yaml
prometheus-adapter-service.yaml
prometheus-clusterRoleBinding.yaml
prometheus-clusterRole.yaml
prometheus-prometheus.yaml
prometheus-roleBindingConfig.yaml
prometheus-roleBindingSpecificNamespaces.yaml
prometheus-roleConfig.yaml
prometheus-roleSpecificNamespaces.yaml
prometheus-rules.yaml
prometheus-serviceAccount.yaml
prometheus-serviceMonitorApiserver.yaml
prometheus-serviceMonitorCoreDNS.yaml
prometheus-serviceMonitorKubeControllerManager.yaml
prometheus-serviceMonitorKubelet.yaml
prometheus-serviceMonitorKubeScheduler.yaml
prometheus-serviceMonitor.yaml
prometheus-service.yaml

3.新建目录重新梳理下

[[email protected] manifests]# mkdir -p operator node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter
[[email protected] manifests]# mv *-serviceMonitor* serviceMonitor/
etheus/[[email protected] manifests]# mv 0prometheus-operator* operator/
[[email protected] manifests]# mv grafana-* grafana/
[[email protected] manifests]# mv kube-state-metrics-* kube-state-metrics/
[[email protected] manifests]# mv alertmanager-* alertmanager/
[[email protected] manifests]# mv node-exporter-* node-exporter/
[[email protected] manifests]# mv prometheus-adapter* adapter/
[[email protected] manifests]# mv prometheus-* prometheus/
[[email protected] manifests]# ls
00namespace-namespace.yaml  alertmanager  kube-state-metrics  operator    serviceMonitor
adapter                     grafana       node-exporter       prometheus
[[email protected] manifests]# ls -lh
total 36K
-rw-r--r-- 1 root root   60 Jun  3 20:05 00namespace-namespace.yaml
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 adapter
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 alertmanager
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 grafana
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 kube-state-metrics
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 node-exporter
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 operator
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 prometheus
drwxr-xr-x 2 root root 4.0K Jun  4 14:23 serviceMonitor

4.部署前注意问题
a.镜像问题
其中k8s.gcr.io/addon-resizer:1.8.4镜像下载不了，需要借助阿里云中转下，其他镜像默认都能下载，如遇到不能下载的也需要中转下再tag到自己私有镜像库

[[email protected]_8_24_centos ~]# docker pull registry.cn-beijing.aliyuncs.com/minminmsn/addon-resizer:1.8.4
1.8.4: Pulling from minminmsn/addon-resizer
90e01955edcd: Pull complete
ab19a0d489ff: Pull complete
Digest: sha256:455eb18aa7a658db4f21c1f2b901c6a274afa7db4b73f4402a26fe9b3993c205
Status: Downloaded newer image for registry.cn-beijing.aliyuncs.com/minminmsn/addon-resizer:1.8.4

[[email protected]_8_24_centos ~]# docker tag registry.cn-beijing.aliyuncs.com/minminmsn/addon-resizer:1.8.4 core-harbor.minminmsn.com/public/addon-resizer:1.8.4
[[email protected]_8_24_centos ~]# docker push core-harbor.minminmsn.com/public/addon-resizer:1.8.4
The push refers to repository [core-harbor.minminmsn.com/public/addon-resizer]
cd05ae2f58b4: Pushed
8a788232037e: Pushed
1.8.4: digest: sha256:455eb18aa7a658db4f21c1f2b901c6a274afa7db4b73f4402a26fe9b3993c205 size: 738

b.访问问题
grafana，prometheus，alermanager等如果不想使用ingres方式访问就需要使用nodeport方式，否则对外不好访问
nodeport方式需在service配置文件，如grafana/grafana-service.yaml 添加type: NodePort,如果要指定node对外端口，需要加配nodePort: 33000，具体可以看配置文件
ingress方式也需要配置文件，ingress配置文件见最后访问配置文件，ingress部署参考k8s集群部署ingress

5.应用部署

[[email protected] manifests]# kubectl apply -f .
namespace/monitoring created

[[email protected] manifests]# kubectl apply -f operator/
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
[[email protected] manifests]# kubectl -n monitoring get pod
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-7cb68545c6-z2kjn   1/1     Running   0          41s

[[email protected] manifests]# kubectl apply -f adapter/
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
[[email protected] manifests]# kubectl apply -f alertmanager/
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
[[email protected] manifests]# kubectl apply -f node-exporter/
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
[[email protected] manifests]# kubectl apply -f kube-state-metrics/
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
role.rbac.authorization.k8s.io/kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
[[email protected] manifests]# kubectl apply -f grafana/
secret/grafana-datasources created
configmap/grafana-dashboard-k8s-cluster-rsrc-use created
configmap/grafana-dashboard-k8s-node-rsrc-use created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pods created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
[[email protected] manifests]# kubectl apply -f prometheus/
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
[[email protected] manifests]# kubectl apply -f serviceMonitor/
servicemonitor.monitoring.coreos.com/prometheus-operator created
servicemonitor.monitoring.coreos.com/alertmanager created
servicemonitor.monitoring.coreos.com/grafana created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
servicemonitor.monitoring.coreos.com/node-exporter created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created

6.检查验证

[[email protected] manifests]# kubectl -n monitoring get all
NAME                                       READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0                    2/2     Running   0          91s
pod/alertmanager-main-1                    2/2     Running   0          74s
pod/alertmanager-main-2                    2/2     Running   0          67s
pod/grafana-fc6fc6f58-22mst                1/1     Running   0          89s
pod/kube-state-metrics-8ffb99887-crhww     4/4     Running   0          82s
pod/node-exporter-925wp                    2/2     Running   0          89s
pod/node-exporter-f45s4                    2/2     Running   0          89s
pod/prometheus-adapter-66fc7797fd-x6l5x    1/1     Running   0          90s
pod/prometheus-k8s-0                       3/3     Running   1          88s
pod/prometheus-k8s-1                       3/3     Running   1          88s
pod/prometheus-operator-7cb68545c6-z2kjn   1/1     Running   0          12m

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/alertmanager-main       ClusterIP   10.254.83.142    <none>        9093/TCP            91s
service/alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   91s
service/grafana                 NodePort    10.254.162.5     <none>        3000:33000/TCP      89s
service/kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   90s
service/node-exporter           ClusterIP   None             <none>        9100/TCP            90s
service/prometheus-adapter      ClusterIP   10.254.123.201   <none>        443/TCP             91s
service/prometheus-k8s          ClusterIP   10.254.51.81     <none>        9090/TCP            89s
service/prometheus-operated     ClusterIP   None             <none>        9090/TCP            89s
service/prometheus-operator     ClusterIP   None             <none>        8080/TCP            12m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
daemonset.apps/node-exporter   2         2         2       2            2           beta.kubernetes.io/os=linux   90s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana               1/1     1            1           89s
deployment.apps/kube-state-metrics    1/1     1            1           90s
deployment.apps/prometheus-adapter    1/1     1            1           91s
deployment.apps/prometheus-operator   1/1     1            1           12m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-fc6fc6f58                1         1         1       89s
replicaset.apps/kube-state-metrics-68865c459c    0         0         0       90s
replicaset.apps/kube-state-metrics-8ffb99887     1         1         1       82s
replicaset.apps/prometheus-adapter-66fc7797fd    1         1         1       91s
replicaset.apps/prometheus-operator-7cb68545c6   1         1         1       12m

NAME                                 READY   AGE
statefulset.apps/alertmanager-main   3/3     91s
statefulset.apps/prometheus-k8s      2/2     89s

7.ingress配置

[[email protected] manifests]# cat ingress-monitor.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-ing
  namespace: monitoring
spec:
  rules:
  - host: prometheus-k8s.minminmsn.com
    http:
      paths:
      - backend:
          serviceName: prometheus-k8s
          servicePort: 9090
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana-ing
  namespace: monitoring
spec:
  rules:
  - host: grafana-k8s.minminmsn.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alertmanager-ing
  namespace: monitoring
spec:
  rules:
  - host: alertmanager-k8s.minminmsn.com
    http:
      paths:
      - backend:
          serviceName: alertmanager-main
          servicePort: 9093

[[email protected] manifests]# kubectl apply -f ingress-monitor.yaml
ingress.extensions/prometheus-ing created
ingress.extensions/grafana-ing created
ingress.extensions/alertmanager-ing created

浏览器访问

1.nodeport方式访问
http://10.2.8.65:33000

2.ingress方式访问
http://grafana-k8s.minminmsn.com
默认账号密码admin admin需要重置密码进入

原文地址：https://blog.51cto.com/jerrymin/2405042

时间： 2024-11-06 03:32:55

kubernetes集群全栈监控报警方案kube-prometheus的相关文章

【prometheus】kubernetes集群性能监控

手动安装方案——阿里云解决方案(部署失败): https://www.jianshu.com/p/1c7ddf18e8b2 手动安装方案—— (部署成功,但是只有CPU内存等的监控信息,没有GPU的监控信息): https://github.com/camilb/prometheus-kubernetes/tree/master helm安装方案——GPU-Monitoring-tools解决方案(部署成功): 参考:http://fly-luck.github.io/2018/12/10/gp

Rancher2.x 一键式部署 Prometheus + Grafana 监控 Kubernetes 集群

目录 1.Prometheus & Grafana 介绍 2.环境.软件准备 3.Rancher 2.x 应用商店 4.一键式部署 Prometheus 5.验证 Prometheus + Grafana 1.Prometheus & Grafana 介绍 Prometheus 是一套开源的系统监控.报警.时间序列数据库的组合,Prometheus 基本原理是通过 Http 协议周期性抓取被监控组件的状态,而通过 Exporter Http 接口输出这些被监控的组件信息,而且已经有很多 E

阿里巴巴大规模神龙裸金属 Kubernetes 集群运维实践

作者 | 姚捷(喽哥)阿里云容器平台集群管理高级技术专家 本文节选自<不一样的 双11 技术:阿里巴巴经济体云原生实践>一书,点击即可完成下载. 导读:值得阿里巴巴技术人骄傲的是 2019 年阿里巴巴 双11?核心系统 100% 以云原生的方式上云,完美支撑了?54.4w 峰值流量以及?2684 亿的成交量.背后承载海量交易的计算力就是来源于容器技术与神龙裸金属的完美融合. 集团上云机器资源形态 阿里巴巴 双11 采用三地五单元架构,除 2 个混部单元外,其他 3 个均是云单元.神龙机型经过

Kubernetes集群

Kubernetes已经成为当下最火热的一门技术,未来一定也会有更好的发展,围绕着云原生的周边产物也越来越多,使得上云更加便利更加有意义,本文主要讲解一些蔚来汽车从传统应用落地到Kubernetes集群的一些实践经验,提供给大家在落地之前的一些思考和注意点,并且让大家在实施的时候能够有一些借鉴,提供一些使用过程中的注意事项.项目背景 Docker诞生于2013年初,随着时间的推移Docker项目也逐渐火热起来,也形成了自己的生态,为了能够灵活调度容器,编排技术也变得非常重要,Swarm,Meso

在 Linux 部署多节点 Kubernetes 集群与 KubeSphere 容器平台

KubeSphere 是在 Kubernetes 之上构建的以应用为中心的企业级容器平台,所有供为用户提供简单易用的操作界面以及向导式操作方式.同时,KubeSphere Installer 提供了 快速部署生产级别的 Kubernetes 集群和全栈化容器平台的功能,能够快速帮助企业用户快速搭建一套以应用为中心的 DevOps 平台. KubeSphere 支持部署和运行在包括公有云.私有云.VM.BM 和 Kubernetes 等任何基础设施之上,可以部署在公有云托管的 Kubernetes

安装部署Kubernetes集群实战

kubernetes概述: Kubernetes是一个开源的,用于管理云平台中多个主机上的容器化的应用,Kubernetes的目标是让部署容器化的应用简单并且高效(powerful),Kubernetes提供了应用部署,规划,更新,维护的一种机制.Kubernetes是Google 2014年创建管理的,是Google 10多年大规模容器管理技术Borg的开源版本. 通过kubernetes可以实现的功能: 快速部署应用 快速扩展应用 无缝对接新的应用功能 节省资源,优化硬件资源的使用 我们的目

CentOS下Kubernetes集群架设（一）主机环境预设

虽然有Rancher OS和CoreOS这类的发行版,但Kubernetes集群的安装也不是太麻烦,因此,还是先从最基本的实验下.以下是本人CentOS7.6上安装Kubernetes集群的笔记,并且分章节记录,持续更新.... Kubernetes主机环境预设 Kubernete集群的主机生产环境也有多种选择,如下: 方案一: 三台或者五台 Master 节点,分别安装全角色:ETCD , Control Plane :其他节点为容器计算机节点,分别安装角色: worker: 方案二: 三台节

Kubernetes集群水平扩展——HPA（自动伸缩）

Kubernetes集群可以通过Replication Controller的scale机制完成服务的扩容或缩容,实现具有伸缩性的服务. Kubernetes集群自动伸缩分为: sacle手动伸缩:可参考K8s资源对象的基本管理之使用命令行的方式(升级.回滚.扩容.缩容): autoscale自动伸缩:也就是本篇博文所介绍的HPA: Kubernetes自动扩展主要分为: 水平扩展:针对实例数目的增减: 垂直扩展:也就是单个实例就可以使用的资源的增减,比如增加CPU.内存: 一.HPA简介 HP

基于Python+Django的Kubernetes集群管理平台

原文出自[听云技术博客]:http://blog.tingyun.com/web/article/detail/1345 时至今日,接触kubernetes也有一段时间了,而我们的大部分业务也已经稳定地运行在不同规模的kubernetes集群上,不得不说,无论是从应用部署.迭代,还是从资源调度管理等方面都有其难以言喻的优势,但是随着业务的不断增长,以及服务的多元化,容器的体量与管理的难度也随之增长. 浅述Kubernetes集群日常管理维护中的一些痛点: 1.较为庞大的集群规模及容器数量维护管理