master上启动的容器一直pending

文章目录

【报错】

  • 隔壁部门研发环境k8s集群（单master）的master上 kube-flannel 和 kube-proxy 一直pending，找我们帮忙排查。

  • describe 查看pod 报错说挂载问题，一个节点上所有pod挂载不上，大概率是kubelet会报错（当然也可以从journalctl命令过滤错误日志）。

  • kubelet 报错如下：

Redirecting to /bin/systemctl status  -l kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 五 2021-10-08 12:35:54 CST; 53min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 51270 (kubelet)
    Tasks: 16
   Memory: 48.3M
   CGroup: /system.slice/kubelet.service
           └─51270 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1

10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.379424   51270 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\" (\"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\")" failed. No retries permitted until 2021-10-08 13:31:53.37940227 +0800 CST m=+3359.052988748 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"flannel-token-bfxdm\" (UniqueName: \"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\") pod \"kube-flannel-ds-amd64-wh59m\" (UID: \"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\") : couldn't propagate object cache: timed out waiting for the condition"
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.382424   51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-token-pnlcd": Failed to list *v1.Secret: secrets "ingress-nginx-token-pnlcd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.580367   51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-admission": Failed to list *v1.Secret: secrets "ingress-nginx-admission" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.780370   51270 reflector.go:125] object-"kube-system"/"flannel-token-bfxdm": Failed to list *v1.Secret: secrets "flannel-token-bfxdm" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.980323   51270 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.980714   51270 reflector.go:125] object-"kube-system"/"kube-flannel-cfg": Failed to list *v1.ConfigMap: configmaps "kube-flannel-cfg" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.986133   51270 kubelet_node_status.go:72] Attempting to register node xl-master
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.380341   51270 reflector.go:125] object-"kube-system"/"kube-proxy-token-74txd": Failed to list *v1.Secret: secrets "kube-proxy-token-74txd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.580340   51270 reflector.go:125] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.780279   51270 reflector.go:125] object-"kubesphere-monitoring-system"/"qingcloud": Failed to list *v1.Secret: secrets "qingcloud" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kubesphere-monitoring-system": no relationship found between node "XL-master" and this object

【分析】

  • 重点是这里
User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object

如上可知 system:node:XL-master用户没有该命名空间的相关权限

  • 配置文件
    查看 /etc/kubernetes/kubelet.conf文件，kubelet 确实使用的是 system:node:XL-master用户。
    具当事部门运维人员说并没有更换过用户，权限绑定也没有人删除过。好吧我们是友军，不深究了。

【解决】

创建一个clusterrolebinding给system:node:XL-master用户绑定cluster-admin权限

kubectl create clusterrolebinding 10-18-XL-master --clusterrole=cluster-admin --user=system:node:XL-master

说明：10-18-XL-master是 clusterrolebinding 的名字，随便起的。

posted on 2021-10-09 10:12  运维开发玄德公  阅读(29)  评论(0)  编辑  收藏  举报  来源

导航