kubernetes 集群中某些 pod 启动失败, 报错 'failed to reserve container name'

问题环境:

系统: Ubuntu 22.04.1 LTS
使用 kubeadm 初始化的 kubernetes 集群
容器运行时: containerd
containerd 版本: containerd containerd.io 1.6.22 8165feabfdfe38c65b599c4993d227328c231fca
Kubelet Version: v1.26.1

完整错误信息:

failed to reserve container name "kube-scheduler_kube-scheduler-laplus-main2_kube-system_352ff1bfb8ed65b7c8d4736f3db2a683_6": name "kube-scheduler_kube-scheduler-laplus-main2_kube-system_352ff1bfb8ed65b7c8d4736f3db2a683_6" is reserved for "7426257f2e40d5ba2eac5f36084a251e81668f0e8776b3af9ce33602311e5906"

这里贴的是 kube-scheduler 这个 pod 的错误, 实际其他少数 pod 也有类似的错误, 都是同一个原因, 相同的方法可以解决

原因以及最后解决方案

强制删除旧的 kube-scheduler 容器

通过下面的命令, 找出旧的 kube-scheduler

crictl ps -a | grep kube-scheduler

我在运行这个命令时遇到了错误

WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
E0209 13:18:16.558238  242920 remote_runtime.go:390] "ListContainers with filter from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" filter="&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},}"
FATA[0000] listing containers: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"

需要手动指定 socket 文件, 加上 --runtime-endpoint 参数即可

crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube-scheduler

命令输出如下:

c3dbf6e96d141       655493523f607       Less than a second ago   Exited              kube-scheduler            5                   1254c254c29fe       kube-scheduler-laplus-main2
7426257f2e40d       655493523f607       3 hours ago              Running             kube-scheduler            6                   1254c254c29fe       kube-scheduler-laplus-main2

删除导致冲突的容器即可, 直接两个都删了也行, 可以看到两个容器的id是 c3dbf6e96d141 和 7426257f2e40d, 下面的命令进行删除操作

crictl --runtime-endpoint unix:///run/containerd/containerd.sock rm c3dbf6e96d141 7426257f2e40d
c3dbf6e96d141

命令输出如下:

ERRO[0000] container "7426257f2e40d" is running, please stop it first
FATA[0000] unable to remove container(s)
laggage@laplus-main2:~$ crictl --runtime-endpoint unix:///run/containerd/containerd.sock ^C426257f2e40d

虽然这里报错说 "7426257f2e40d" 这个容器正在运行, 无法删除, 但是实际再去检查一下, 对应的 pod 已经正常启动不在报 'failed to reserve container name' 这个错误了, 也就是说删除第一个 c3dbf6e96d141 已经解决问题了, 皆大欢喜~

本次排查耗时 2 小时左右, 特此记录下来, 看到的朋友如果对你有启发或帮助, 不妨动手点个赞~