k8s集群证书过期问题处理

1.问题出现

   因为临时想在kubernetes集群(测试环境)上创建Pod发现好久没用得k8s集群无法使用，报错如下:

   kubectl get ns

   The connection to the server 10.21.4.113:6443 was refused - did you specify the right host or port?  

   就想到是kube-spiserver出了问题。所以就使用systemctl status kubelet -l 查看具体原因:

   

   systemctl status kubelet.service
  ● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since 二 2021-12-07 15:20:44 CST; 35s ago
Docs: https://kubernetes.io/docs/
Main PID: 28356 (kubelet)
Tasks: 55
Memory: 48.5M
CGroup: /system.slice/kubelet.service
└─28356 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kub...

12月 07 15:21:18 k8smaster kubelet[28356]: E1207 15:21:18.985804 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.086034 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.186265 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.286503 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.386722 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.486971 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.587204 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.687460 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.787686 28356 kubelet.go:2263] node "k8smaster" not found
12月 07 15:21:19 k8smaster kubelet[28356]: E1207 15:21:19.888025 28356 kubelet.go:2263] node "k8smaster" not found

根据报错，认为两个组件出现了问题，要么是kube-apiserver，要么就是etcd出了问题。

2.处理问题

因为我的测试k8s集群是由kubeadm安装的，所以我就使用docker ps 还有docker logs查看kube-apiserver和etcd的相关log。看看问题到底出现在哪里

 

 可以看出来k8s集群得证书过期了，然后更新证书就可以了

Kubelet组件证书默认有效期为1年。集群运行1年以后就会导致报 certificate has expired or is not yet valid 错误，导致集群 Node不能于集群 Master正常通信。重启的话k8s就起不来了。

重新生成证书
验证证书是否过期：

openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt

1.14后的版本可以使用这个命令查看过期时间，如下：

kubeadm alpha certs check-expiration

kubeadm 安装得证书默认为 1 年，注意原证书文件必须保留在服务器上才能做延期操作，否则就会重新生成，集群可能无法恢复。
先把原配置和证书备份

cp -rp /etc/kubernetes /etc/kubernetes.bak

如果 kubeadm配置文件找不到了，就先生成一个默认的，然后自行修改：

kubeadm config print init-defaults > kubeadm.yaml

然后根据你自己的实际情况修改：主要修改：kubernetesVersion、advertiseAddress、imageRepository、serviceSubnet

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.21.4.113
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8smaster
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}

修改完，用以上配置重新生成证书：

kubeadm alpha certs renew all --config=/data/kubeadm.yaml

延期配置之后需要更新配置文件

# 注意：更新配置文件前先以 move 方式备份，或删除配置文件
mv /etc/kubernetes/*.conf /data/kubeconfback/
kubeadm init phase kubeconfig all --config=/data/kubeadm.yaml

之后重启 kube-apiserver,etcd,scheduler,controller 容器

docker ps | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | awk '{print $1}' | awk '{print "docker","restart",$1}' | bash

或者重启 kubelet

systemctl restart kubelet

 

如果重启kubelet后发现出现这样得报错  error: You must be logged in to the server (Unauthorized)。还得更改下权限(因为重新为组件分配了证书)

 echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile                 

source ~/.bash_profile

如果是非root用户:

mkdir -p $HOME/.kube          

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config         

sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

最后查看集群状态：