k8s常见异常

1、证书过期

[root@kube-master01 kubernetes]# kubectl get node
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-03-19T09:30:10+08:00 is after 2023-01-24T09:34:35Z

[root@kube-master01 log]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 24, 2023 09:34 UTC   <invalid>       ca                      no
apiserver                  Jan 24, 2023 09:34 UTC   <invalid>       ca                      no
apiserver-etcd-client      Jan 24, 2023 09:34 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Jan 24, 2023 09:34 UTC   <invalid>       ca                      no
controller-manager.conf    Jan 24, 2023 09:34 UTC   <invalid>       ca                      no
etcd-healthcheck-client    Jan 24, 2023 09:34 UTC   <invalid>       etcd-ca                 no
etcd-peer                  Jan 24, 2023 09:34 UTC   <invalid>       etcd-ca                 no
etcd-server                Jan 24, 2023 09:34 UTC   <invalid>       etcd-ca                 no
front-proxy-client         Jan 24, 2023 09:34 UTC   <invalid>       front-proxy-ca          no
scheduler.conf             Jan 24, 2023 09:34 UTC   <invalid>       ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 22, 2032 09:34 UTC   8y              no
etcd-ca                 Jan 22, 2032 09:34 UTC   8y              no
front-proxy-ca          Jan 22, 2032 09:34 UTC   8y              no

[root@kube-master01 log]# date
Sun Mar 19 09:00:21 CST 2023

可以通过 kubeadm certs renew 证书名，更新对应证书，

[root@kube-master01 log]# kubeadm certs renew --help
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm certs renew [flags]
  kubeadm certs renew [command]

Available Commands:
  admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
  all                      Renew all available certificates
  apiserver                Renew the certificate for serving the Kubernetes API
  apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd
  apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
  controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use
  etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd
  etcd-peer                Renew the certificate for etcd nodes to communicate with each other
  etcd-server              Renew the certificate for serving etcd
  front-proxy-client       Renew the certificate for the front proxy client
  scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

由于上述过期证书太多，使用命令：kubeadm certs renew all，更新所有证书

更新前最好备份/etc/kubernetes/*.conf 文件

[root@kube-master01 log]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

 

2、在 kubernetes control plane 服务器上运行 kubeadm certs renew all 命令更新证书后，kubelet 无法正常启动，syslog 在报错信息如下：

"Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

对于高可用集群，需要加上 control-plane-endpoint 参数

$ kubeadm init phase certs all --control-plane-endpoint "k8s-api:6443"
$ kubeadm init phase kubeconfig all --control-plane-endpoint "k8s-api:6443"