Prometheus学习笔记之微服务kube-state-metrics报错

0x00 概述

在K8S集群部署kube-state-metrics微服务的时候，发现容器日志不停刷报错日志，主要报错日志如下：

E0824 13:09:36.768882 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list secrets at the cluster scope
E0824 13:09:36.742450 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list jobs.batch at the cluster scope
E0824 13:09:36.743385 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list poddisruptionbudgets.policy at the cluster scope
E0824 13:09:36.568839 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list endpoints at the cluster scope
E0824 13:09:36.379898 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list configmaps at the cluster scope
E0824 13:09:36.317600 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v2beta1.HorizontalPodAutoscaler: horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list horizontalpodautoscalers.autoscaling at the cluster scope
E0824 13:09:36.316554 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list statefulsets.apps at the cluster scope
E0824 13:09:36.318569 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1beta1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list cronjobs.batch at the cluster scope
E0824 13:09:35.768772 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list namespaces at the cluster scope
E0824 13:09:36.168855 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list persistentvolumes at the cluster scope
E0824 13:09:35.742782 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list poddisruptionbudgets.policy at the cluster scope
E0824 13:09:35.568827 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list secrets at the cluster scope
E0824 13:09:35.741814 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list jobs.batch at the cluster scope
E0824 13:09:35.968853 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list persistentvolumeclaims at the cluster scope
E0824 13:09:35.318064 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1beta1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list cronjobs.batch at the cluster scope
E0824 13:09:35.368786 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/builder.go:508: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list endpoints at the cluster scope

发现是kube-state-metrics在集群权限不足；

在github上下载的yaml文件在执行clusterrolebing那一步，并没有给kube-state-metrics提供cluster层级的权限；

 

0x02 给kube-state-metrics赋权cluster-admin

执行如下命令，给system:serviceaccount:monitoring:kube-state-metrics做clusterrolebing

kubectl create clusterrolebinding kube-state-metrics-admin-binding \
--clusterrole=cluster-admin  \
--user=system:serviceaccount:monitoring:kube-state-metrics