Kubernetes heapster

Heapster是容器集群监控和性能分析工具，HPA、Dashborad、Kubectl top都依赖于heapster收集的数据。

但是Heapster从kubernetes 1.8以后已经被遗弃了...... 被metrics-server所替代......

kubernetes 1.11 中部署Heapster 1.5.4版本的过程

Heapster 部署yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: heapster
  namespace: kube-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        # image: k8s.gcr.io/heapster-amd64:v1.5.4 将默认google的官方镜像替换为阿里云镜像，否则你懂得
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: ‘true‘
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

heapster.yaml

heapster 启动参数说明：

  • inClusterConfig - Use kube config in service accounts associated with Heapster‘s namespace. (default: true)
  • kubeletPort - kubelet port to use (default: 10255)
  • kubeletHttps - whether to use https to connect to kubelets (default: false)
  • insecure - whether to trust Kubernetes certificates (default: false)
  • auth - client auth file to use. Set auth if the service accounts are not usable.
  • useServiceAccount - whether to use the service account token if one is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token (default: false)

使用： kubectl apply -f heapster.yaml 部署

部署后查看log日志，发现一直提示“403 Forbidden”, response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"

[[email protected] heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-855k7
I0912 14:36:48.175115       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
I0912 14:36:48.175174       1 heapster.go:79] Heapster version v1.5.4
I0912 14:36:48.175366       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0912 14:36:48.175393       1 configs.go:62] Using kubelet port 10250
I0912 14:36:48.185828       1 heapster.go:202] Starting with Metric Sink
I0912 14:36:48.199515       1 heapster.go:112] Starting heapster on port 8082
E0912 14:37:05.000327       1 kubelet.go:288] node node01 is not ready
E0912 14:38:05.016044       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:38:25.000643       1 manager.go:152] Failed to get all responses in time (got 0/1)
E0912 14:39:05.008534       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:39:25.000674       1 manager.go:152] Failed to get all responses in time (got 0/1)
E0912 14:40:05.009955       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:40:25.001231       1 manager.go:152] Failed to get all responses in time (got 0/1)
E0912 14:41:05.017198       1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0912 14:41:25.000846       1 manager.go:152] Failed to get all responses in time (got 0/1)

查看ClusterRole: system:heapster的权限，发现的确没有针对Resource: nodes/stats 的create权限

[[email protected] heapster-yaml]# kubectl describe clusterrole system:heapster
Name:         system:heapster
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"lab...
              rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources               Non-Resource URLs  Resource Names  Verbs
  ---------               -----------------  --------------  -----
  events                  []                 []              [get list watch]
  namespaces              []                 []              [get list watch]
  nodes                   []                 []              [get list watch]
  pods                    []                 []              [get list watch]
  deployments.extensions  []                 []              [get list watch]

修改ClusterRole: system:heapster的权限:

1. 查看system:heapster yaml格式, 保存为 heapster-clusterrole.yaml

yaml

[[email protected] heapster-yaml]# kubectl get clusterrole system:heapster -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"labels":{"kubernetes.io/bootstrapping":"rbac-defaults"},"name":"system:heapster","namespace":""},"rules":[{"apiGroups":[""],"resources":["events","namespaces","nodes","pods","nodes/stats"],"verbs":["create","get","list","watch"]},{"apiGroups":["extensions"],"resources":["deployments"],"verbs":["get","list","watch"]}]}
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2018-08-26T02:26:14Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:heapster
  resourceVersion: "139000"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Aheapster
  uid: 67ef3689-a8d7-11e8-a891-000c29b52823
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

2. 添加Resource: nodes/stats的create权限，并执行 kubectl apply -f heapster-clusterrole.yaml

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:heapster
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  - pods
  - nodes/stats
  verbs:
  - create
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

3. 删除heapster重新部署

　　kubectl delete -f heapster.yaml

　　kubectl apply -f heapster.yaml

4. 重新部署后，查看log没有任何报错了

[[email protected] heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-8zgxq
I0912 15:02:38.926068       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
I0912 15:02:38.926128       1 heapster.go:79] Heapster version v1.5.4
I0912 15:02:38.926565       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0912 15:02:38.926647       1 configs.go:62] Using kubelet port 10250
I0912 15:02:38.938333       1 heapster.go:202] Starting with Metric Sink
I0912 15:02:38.949215       1 heapster.go:112] Starting heapster on port 8082

5. 功能测试

部署前：

[[email protected] heapster-yaml]# kubectl top pod
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

部署后：

[[email protected] tomcat-mysql-yaml]# kubectl top pod
NAME                      CPU(cores)   MEMORY(bytes)
lxcfs-j8bpd               0m           4Mi
mysql-85c85c5668-fr6j7    0m           458Mi
tomcat-78c9778858-424gc   1m           93Mi
tomcat-78c9778858-42xz4   1m           97Mi
tomcat-78c9778858-6t7cd   1m           93Mi

相关链接：

https://github.com/kubernetes/heapster

https://github.com/kubernetes-incubator/metrics-server

https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/

原文地址：https://www.cnblogs.com/vincenshen/p/9638162.html

时间： 2024-10-10 19:00:32

Kubernetes heapster的相关文章

Kubernetes 1.5集成heapster

Heapster是kubernetes集群监控工具.在1.2的时候,kubernetes的监控需要在node节点上运行cAdvisor作为agent收集本机和容器的资源数据,包括cpu.内存.网络.文件系统等.在新版的kubernetes中,cAdvisor被集成到kubelet中.通过netstat可以查看到kubelet新开了一个4194的端口,这就是cAdvisor监听的端口,现在我们然后可以通过http://<node-ip>:4194的方式访问到cAdvisor.Heapster就是

Kubernetes集成Heapster监控

Heapster是容器集群监控和性能分析工具,天然的支持Kubernetes和CoreOS.Kubernetes有个出名的监控agent-cAdvisor.在每个kubernetes Node上都会运行cAdvisor,它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime).在较新的版本中,K8S已经将cAdvisor功能集成到kubelet组件中.每个Node节点可以直接进行web访问. 1.镜像下载[[email protected]_ma

详解k8s一个完整的监控方案(Heapster+Grafana+InfluxDB) - kubernetes

1.浅析整个监控流程 heapster以k8s内置的cAdvisor作为数据源收集集群信息,并汇总出有价值的性能数据(Metrics):cpu.内存.网络流量等,然后将这些数据输出到外部存储,如InfluxDB,最后就可以通过相应的UI界面显示出来,如grafana. 另外heapster的数据源和外部存储都是可插拔的,所以可以很灵活的组建出很多监控方案,如:Heapster+ElasticSearch+Kibana等等. 2.创建k8s资源对象 使用官方提供的yml文件有一些小问题,请参考以下

使用 kubeadm 安装部署 kubernetes 1.9-部署heapster插件

1.先到外网下载好镜像倒进各个节点 2.下载yaml文件和创建应用 mkdir -p ~/k8s/heapster cd ~/k8s/heapster wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/d

kubernetes 监控方案之：heapster+influxdb+grafana(十八)

目录 一.Heapster 介绍 二.部署 三.使用 heapster已经deprecated了:https://github.com/kubernetes/heapster 一.Heapster 介绍 Heapster 是容器集群监控和性能分析工具,天然的支持 Kubernetes 和 CoreOS. Kubernetes 有个出名的监控 agent-cAdvisor.在每个 kubernetes Node 上都会运行 cAdvisor,它会收集本机以及容器的监控数据 (cpu,memory,

k8s监控组件heapster安装部署

k8s监控组件heapster安装部署 参考文档 https://github.com/kubernetes/heapster/tree/master/deploy k8s集群安装部署 http://jerrymin.blog.51cto.com/3002256/1898243  k8s集群RC.SVC.POD部署 http://jerrymin.blog.51cto.com/3002256/1900260     k8s集群组件kubernetes-dashboard和kube-dns部署 h

Kubernetes资源监控探索

搭建kubernetes集群,有一个默认的dashboard,但是这个dashboard比较简陋,不能将自定义展示.所以打算使用Grafana+Heapster+Influxdb构建一个一体化监控平台 首先呢?从https://github.com/kubernetes/heapster获取源码,在heapster/deploy/kube-config/influxdb文件夹下将所有的yaml文件,使用kubectl create,这样就分别的到 了heapster,grafana,influx

Kubernetes1.5 集成Heapster

Kubernetes1.5 集成Heapster Heapster是kubernetes集群监控工具.在1.2的时候,kubernetes的监控需要在node节点上运行cAdvisor作为agent收集本机和容器的资源数据,包括cpu.内存.网络.文件系统等.在新版的kubernetes中,cAdvisor被集成到kubelet中.通过netstat可以查看到kubelet新开了一个4194的端口,这就是cAdvisor监听的端口,现在我们然后可以通过http://<node-ip>:4194

Kubernetes插件部署

1. 部署 KubeDNS 插件 官方的配置文件中包含以下镜像: kube-dns          ----监听service.pod等资源,动态更新DNS记录 sidecar           ----用于监控和健康检查 dnsmasq           ----用于缓存,并可从dns服务器获取dns监控指标 地址: https://github.com/kubernetes/dns 官方的yaml文件目录:kubernetes/cluster/addons/dns https://gi