k8s 证书更新，基于kubernetes v1.19.3

学习记录：

通过kubeadm安装的K8S集群，证书有效期为一年，一年过期后，会导致api service不可用，使用过程中会出现报错：x509: certificate has expired or is not yet valid.

目前证书更新的方法：

官方推荐一年之内使用kubeadm upgrade更新一次kubernetes系统。   
源代码编译安装，使得证书的时间延长。
一年内手动更新证书。
启用自动轮换kubelet证书。

重点记录3、4

在master上使用如下命令查看证书过期时间

[root@master]# kubeadm alpha certs check-expiration
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 19, 2021 09:53 UTC   334d                                    no      
apiserver                  Oct 19, 2021 09:52 UTC   334d            ca                      no      
apiserver-etcd-client      Oct 19, 2021 09:53 UTC   334d            etcd-ca                 no      
apiserver-kubelet-client   Oct 19, 2021 09:52 UTC   334d            ca                      no      
controller-manager.conf    Oct 19, 2021 09:53 UTC   334d                                    no      
etcd-healthcheck-client    Oct 19, 2021 09:53 UTC   334d            etcd-ca                 no      
etcd-peer                  Oct 19, 2021 09:53 UTC   334d            etcd-ca                 no      
etcd-server                Oct 19, 2021 09:53 UTC   334d            etcd-ca                 no      
front-proxy-client         Oct 19, 2021 09:52 UTC   334d            front-proxy-ca          no      
scheduler.conf             Oct 19, 2021 09:53 UTC   334d                                    no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 17, 2030 09:52 UTC   9y              no      
etcd-ca                 Oct 17, 2030 09:53 UTC   9y              no      
front-proxy-ca          Oct 17, 2030 09:52 UTC   9y              no

一、手动更新证书。（证书还未过期的情况下）

1. 集群还能访问的情况下，使用： kubeadm config view > kube-config.yaml 生成集群的yaml文件，可以提前准备好免得集群挂掉之后不能生成。

[root@master ~]# cd /etc/kubernetes/manifests/
[root@master manifests]# kubeadm config view > kube-config.yaml
[root@master manifests]# cat kube-config.yaml 
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.19.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.1.0.0/16
scheduler: {}

　　2. 备份原有的证书文件

1	`[root@master manifests]# cp -r /etc/kubernetes/pki/ /etc/kubernetes/pki_backup`

　　3. 开始更新证书

1	`[root@master manifests]# kubeadm alpha certs renew all --config=kube-config.yaml`

4. 完成后重启master上kube-apiserver,kube-controller,kube-scheduler,etcd这4个容器，如果有多台master，则将第一台生成的相关证书拷贝到其余master即可

[root@master pki]# docker restart `docker ps | grep etcd | awk '{print $1}'`
8b09bcb64cd0
eb63e6c341e4
[root@master pki]# docker restart `docker ps | grep kube-apiserver | awk '{print $1}'`
6d8afc50d03a
84261c9cb25f
[root@master pki]# docker restart `docker ps | grep kube-controller | awk '{print $1}'`
ba3cc2a57987
[root@master pki]# docker restart `docker ps | grep kube-scheduler | awk '{print $1}'`
5fd115b29da1
8011162e1cc8

　　5. 查看pod集群状态，检查刚刚重启的status是否为Running(一般会等待2分钟左右)

1	`kubectl get pods --all-namespaces -o wide`

　　6. 查看当前集群的证书时间，RESIDUAL TIME，为364d，续期一年。

[root@master pki]# kubeadm alpha certs check-expiration
 
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Nov 19, 2021 08:47 UTC   364d                                    no      
apiserver                  Nov 19, 2021 08:47 UTC   364d            ca                      no      
apiserver-etcd-client      Nov 19, 2021 08:47 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Nov 19, 2021 08:47 UTC   364d            ca                      no      
controller-manager.conf    Nov 19, 2021 08:47 UTC   364d                                    no      
etcd-healthcheck-client    Nov 19, 2021 08:47 UTC   364d            etcd-ca                 no      
etcd-peer                  Nov 19, 2021 08:47 UTC   364d            etcd-ca                 no      
etcd-server                Nov 19, 2021 08:47 UTC   364d            etcd-ca                 no      
front-proxy-client         Nov 19, 2021 08:47 UTC   364d            front-proxy-ca          no      
scheduler.conf             Nov 19, 2021 08:47 UTC   364d                                    no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 17, 2030 09:52 UTC   9y              no      
etcd-ca                 Oct 17, 2030 09:53 UTC   9y              no      
front-proxy-ca          Oct 17, 2030 09:52 UTC   9y              no 

　　# 当前通过手动的方式更新证书完成

启动自动轮换kubelet证书

kubelet 证书分为 server 和 client 两种，kubernetes 1.9 开始默认启用了client证书的自动轮换，server证书的自动轮换还需要我们手动开启。(证书的位置)

[root@master ~]# cd /etc/kubernetes/pki
[root@master pki]# ls
apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  etcd                front-proxy-ca.key      front-proxy-client.key  sa.pub
apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key
[root@master pki]# cd etcd/
[root@master etcd]# ls
ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key  peer.crt  peer.key  server.crt  server.key

　　1. 增加kubelet参数

1 2	`[root@master]# vim /etc/sysconfig/kubelet` `KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true` `--rotate-server-certificates=true`

　　2. 增加kube-controller-manager.yaml 参数

[root@master]# cd /etc/kubernetes/manifests/
[root@master manifests]# vim kube-controller-manager.yaml
 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --experimental-cluster-signing-duration=87600h0m0s #增加证书颁发时间参数
    - --feature-gates=RotateKubeletServerCertificate=true #开启server证书签发
    - --allocate-node-cidrs=true
…………………………
…………………………

　　3. 创建rbac ( rbac是kubernetes的一种认证访问授权机制 ) 对象，允许节点轮换kubelet server证书

cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeserver
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-certificate-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
EOF

4. 根据yaml文件开启server证书自动轮换

1 2	`[root@master manifests]# kubectl apply -f ca-update.yaml` `clusterrolebinding.rbac.authorization.k8s.io/kubeadm:node-autoapprove-certificate-server created`

　　5. 重启kubelet