Failed to list *v1.CronJob: the server could not find the requested resource

基础环境

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:40:09Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.15-eks-84b4fe6", GitCommit:"c813d094d1e9fa409079753049cfaa9a67524ed2", GitTreeState:"clean", BuildDate:"2022-06-09T18:20:31Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Prometheus 告警信息

{container="kube-rbac-proxy-self", endpoint="https-self", instance="10.20.108.17:9443", job="kube-state-metrics", namespace="monitoring", pod="kube-state-metrics-695cc54b57-5nkv2", resource="*v1.CronJob", result="error", service="kube-state-metrics"}
0.00792311111111111
{container="kube-rbac-proxy-self", endpoint="https-self", instance="10.20.108.17:9443", job="kube-state-metrics", namespace="monitoring", pod="kube-state-metrics-695cc54b57-5nkv2", resource="*v1.PodDisruptionBudget", result="error", service="kube-state-metrics"}
0.00792311111111111

问题分析

直接查看 k8s 中 kube-state-metrics pod 的报错；
报错如下：

W0815 04:57:34.763696       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1.PodDisruptionBudget: the server could not find the requested resource
E0815 04:57:34.763739       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: the server could not find the requested resource
W0815 04:57:42.762271       1 refl

问题排查

网上找了不少文档，google了不少材料，发现一个比较有价值的链接：
https://github.com/kubernetes/kube-state-metrics/issues/295
https://github.com/kubernetes/kube-state-metrics/issues/295

按照文档中所说，修改了 sa 权限，然后指定 监控指标。（去除 告警的指标）
但是还是不好使

问题解决

这套监控使用 https://github.com/prometheus-operator/prometheus-operator 来部署的；
安装完后，并没有做定制化的修改；
经最后查看，是 kubeStateMetrics 的版本 和 k8s 版本不兼容，导致有些功能不能支持；
对应的版本信息如下：

线上环境使用的版本如下：

最后修改 kube-state-metrics 的版本为 v2.3.0，然后重新部署，问题解决；