记一次k8s问题处理 => 6443端口没有监听 => 证书过期处理
原始的问题是这样：
The connection to the server 192.168.122.200:6443 was refused - did you specify the right host or port?
lsof -i :6443
端口没有监听
master 的kubelet.service 是active的，docker容器api_server是down的
node的 kubelet.service 是 dead, docker容器都是down的
journalctl -xefu kubelet 看到的日志都是连不上6443端口
后面找到容器kube-apiserver查看日志：
似乎是证书过期了。。。
以下是解决步骤：
一. 检查证书是否过期。
可以通过下面两种方式检查 Kubernetes 的证书是否过期。
1. kubeadm 命令查看
可以通过 kubeadm alpha certs check-expiration 命令查看相关证书是否过期。
[root@master ~]# kubeadm alpha certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jul 08, 2023 14:11 UTC 364d no apiserver Jul 08, 2023 14:11 UTC 364d ca no apiserver-etcd-client Jul 08, 2023 14:11 UTC 364d etcd-ca no apiserver-kubelet-client Jul 08, 2023 14:11 UTC 364d ca no controller-manager.conf Jul 08, 2023 14:11 UTC 364d no etcd-healthcheck-client Jul 08, 2023 14:11 UTC 364d etcd-ca no etcd-peer Jul 08, 2023 14:11 UTC 364d etcd-ca no etcd-server Jul 08, 2023 14:11 UTC 364d etcd-ca no front-proxy-client Jul 08, 2023 14:11 UTC 364d front-proxy-ca no scheduler.conf Jul 08, 2023 14:11 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jul 02, 2031 14:04 UTC 8y no etcd-ca Jul 02, 2031 14:04 UTC 8y no front-proxy-ca Jul 02, 2031 14:04 UTC 8y no
2. openssl 命令查看
版本过低无法使用 kubeadm 命令时，可以通过 openssl 查看对应证书是否过期。
[root@master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not ' Not Before: Jul 4 14:04:27 2021 GMT Not After : Jul 8 14:11:17 2023 GMT
二. 自动更新证书
Kubenetes 在升级控制面板相关组件时会主动更新证书，因此如果保证 Kubernetes 能够定期（一年以内）升级的话，证书会自动更新。
三. 手动更新证书
1. 证书备份
cp -rp /etc/kubernetes /etc/kubernetes.bak
2. 删除旧的证书
将 /etc/kubernetes/pki 下要重新生成的证书删除
sudo rm -rf /etc/kubernetes/pki/apiserver.key
3. 重新生成证书
主要通过 kubeadm alpha certs renew 命令生成，命令简介如下
kubeadm alpha certs renew Usage: kubeadm alpha certs renew [flags] kubeadm alpha certs renew [command] Available Commands: all renew all available certificates apiserver Generates the certificate for serving the Kubernetes API apiserver-etcd-client Generates the client apiserver uses to access etcd apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet etcd-healthcheck-client Generates the client certificate for liveness probes to healtcheck etcd etcd-peer Generates the credentials for etcd nodes to communicate with each other etcd-server Generates the certificate for serving etcd front-proxy-client Generates the client for the front proxy
重新生成所有证书
[root@master ~]# kubeadm alpha certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [renew] Error reading configuration from the Cluster. Falling back to default configuration W0708 11:36:20.404836 6022 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed
4. 重新生成配置文件
备份旧的配置
mv /etc/kubernetes/*.conf /tmp/
生成新的配置
主要通过 kubeadm init phase kubeconfig 命令执行：
kubeadm init phase kubeconfig -h Usage: kubeadm init phase kubeconfig [flags] kubeadm init phase kubeconfig [command] Available Commands: admin Generates a kubeconfig file for the admin to use and for kubeadm itself all Generates all kubeconfig files controller-manager Generates a kubeconfig file for the controller manager to use kubelet Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes scheduler Generates a kubeconfig file for the scheduler to use
5. 后续操作
完成证书和配置文件的更新后，需要进行一系列后续操作保证更新生效，主要包括重启 kubelet、更新管理配置。
- 重启 kubelet
systemctl restart kubelet
- 更新 admin 配置
将新生成的 admin.conf 文件拷贝，替换 ~/.kube 目录下的 config 文件。
cp /etc/kubernetes/admin.conf ~/.kube/config
至此master节点就恢复好了
恢复node节点
关于token失效, 重新生成
master集群初始化后，token24小时后就会失效，如果到了token失效时间，node再加入集群，需要重新生产token：
[root@master ~]# kubeadm token list
[root@master ~]#
token没有了
重新生产token
[root@master ~]# kubeadm token create W0708 11:48:22.716513 15630 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] aju641.rs9sumbg24v80hld
获取--discovery-token-ca-cert-hash值
[root@master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //' bb184b9cab8543539620b7736017514058ffe925ea04a8bed4c38465aaa004ae
加入集群命令--在node节点执行
kubeadm join 192.168.122.200:6443 --token gkdzsa.xhh13svso84zie2p \ --discovery-token-ca-cert-hash sha256:bb184b9cab8543539620b7736017514058ffe925ea04a8bed4c38465aaa004ae
重启node的kubelet服务
systemctl restart kubelet
https://blog.csdn.net/a1308422754/article/details/107157009/
https://blog.csdn.net/Ahri_J/article/details/107466921
