问题记录：Minikube重启K8S集群v1.24.0+发生Initial timeout问题

问题描述

执行minikube stop && minikube start报错。报错信息如下：

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
	- 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock logs CONTAINERID'

stderr:
W0215 03:40:32.375153   99610 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

提示kubelet是没有运行的，但是通过systemctl status kubelet发现进程是正常运行的，查看kubelet日志发现：

[root@minikube ~]# journalctl -xeu kubelet |more
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.149848   64597 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandb
ox\" for \"etcd-minikube_kube-system(a75b77b0a9e517a0cac04559a1c583ec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-minikub
e_kube-system(a75b77b0a9e517a0cac04559a1c583ec)\\\": rpc error: code = Unknown desc = failed pulling image \\\"registry.k8s.io/pause:3.6\\\": Error resp
onse from daemon: Head \\\"https://asia-northeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6\\\": dial tcp 142.250.157.82:443: con
nect: connection refused\"" pod="kube-system/etcd-minikube" podUID=a75b77b0a9e517a0cac04559a1c583ec
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.230355   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.330444   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.430833   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.531775   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.632183   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.733334   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.834118   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.935064   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:33 minikube kubelet[64597]: E0215 05:14:33.035527   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:33 minikube kubelet[64597]: E0215 05:14:33.135691   64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"

原因分析

是由于v1.24.0后启用了CRI sandbox(pause) image的配置支持。通过kubeadm init –image-repository设置的镜像地址，不再会传递给cri运行时去下载pause镜像。而是需要在cri运行时的配置文件中设置。

问题处理

使用VPN网络FQ后拉取镜像会成功。或配置CRI运行时设置使用的国内镜像即可：

[root@localhsot ~]# vim /etc/containerd/config.toml
# 追加以下内容
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.aliyuncs.com/k8sxio/pause:3.6"
[root@localhsot ~]# systemctl restart kubelet
[root@localhsot ~]# minikube start