解决kubeadm部署的k8s 1.18.3 集群证书过期
K8S CA证书是10年，但是组件证书的日期只有1年，为了证书一直可用状态需要更新，目前主流的一共有3种：
1、版本升级，只要升级就会让各个证书延期1年，官方设置1年有效期的目的就是希望用户在一年内能升级1次；
2、通过命令续期 （这种只能延长一年）；
3、编译源码Kubeadm，证书有效期可自定义；
本实验环境是单master集群环境，如果是多master集群环境那么需要将master上更新的证书分发到各个节点上！
此文档采用K8s 1.18.3版本，不保证其他版本也适用，建议自行测试。
一、查看证书过期时间
1.1 方式一
$ kubeadm alpha certs check-expiration
个人搭建的集群不知道为什么，使用上述命令无法查看ca的证书有效期，所以记录上方式二！
1.2 方式二
$ for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;
do openssl x509 -in $item -text -noout| grep Not;
echo ======================$item===============;
done
也可以一个一个的进行查看：
$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
二、通过命令续期
2.1 修改集群内所有机器的时间，模拟证书在过期的边缘
$ date -s "2022-3-1 12:00"
2.2 查看证书有效期
为了更直观的看到证书的有效期！
$ kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Mar 03, 2022 16:02 UTC 2d no
apiserver Mar 03, 2022 16:02 UTC 2d ca no
apiserver-etcd-client Mar 03, 2022 16:02 UTC 2d etcd-ca no
apiserver-kubelet-client Mar 03, 2022 16:02 UTC 2d ca no
controller-manager.conf Mar 03, 2022 16:02 UTC 2d no
etcd-healthcheck-client Mar 03, 2022 16:02 UTC 2d etcd-ca no
etcd-peer Mar 03, 2022 16:02 UTC 2d etcd-ca no
etcd-server Mar 03, 2022 16:02 UTC 2d etcd-ca no
front-proxy-client Mar 03, 2022 16:02 UTC 2d front-proxy-ca no
scheduler.conf Mar 03, 2022 16:02 UTC 2d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Mar 01, 2031 16:02 UTC 9y no
etcd-ca Mar 01, 2031 16:02 UTC 9y no
front-proxy-ca Mar 01, 2031 16:02 UTC 9y no
如果证书过期的话，就会出现以下情况：
$ kubectl get pod -n kube-system
Unable to connect to the server: x509: certificate has expired or is not yet valid
2.3 备份原有数据
$ kubeadm config view > /root/kubeadm.yaml
$ cat /root/kubeadm.yaml
apiServer:
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.18.3
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
2.4 备份证书
备份主要就是为了升级失败之后，便于回滚！
$ cp -rp /etc/kubernetes /etc/kubernetes_$(date +%F)
$ ls /etc/kubernetes_2022-03-01/
admin.conf controller-manager.conf kubelet.conf manifests pki scheduler.conf
2.5 更新证书
$ kubeadm alpha certs renew all --config=/root/kubeadm.yaml
2.6 确认证书有效期
$ kubeadm alpha certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
admin.conf Mar 01, 2023 04:02 UTC 364d no
apiserver Mar 01, 2023 04:02 UTC 364d no
apiserver-etcd-client Mar 01, 2023 04:02 UTC 364d no
apiserver-kubelet-client Mar 01, 2023 04:02 UTC 364d no
controller-manager.conf Mar 01, 2023 04:02 UTC 364d no
etcd-healthcheck-client Mar 01, 2023 04:02 UTC 364d no
etcd-peer Mar 01, 2023 04:02 UTC 364d no
etcd-server Mar 01, 2023 04:02 UTC 364d no
front-proxy-client Mar 01, 2023 04:02 UTC 364d no
scheduler.conf Mar 01, 2023 04:02 UTC 364d no
2.7 更新kubeconfig文件
$ rm -f /etc/kubernetes/*.conf
$ kubeadm init phase kubeconfig all --config /root/kubeadm.yaml
2.8 更新客户端证书
$ cp $HOME/.kube/config{,.default}
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ chown $(id -u):$(id -g) $HOME/.kube/config
2.9 重启相关的pod
$ docker ps |egrep "k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd" | awk '{print $1}' | xargs docker rm -f
也可以干脆一点，直接重启docker即可！
2.10 查看pod是否运行正常
$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-8lq2k 1/1 Running 1 363d
kube-system coredns-58cc8c89f4-hz774 1/1 Running 1 363d
kube-system etcd-k8s-master 1/1 Running 1 363d
kube-system kube-apiserver-k8s-master 1/1 Running 1 363d
kube-system kube-controller-manager-k8s-master 1/1 Running 1 363d
kube-system kube-flannel-ds-amd64-fh9nx 1/1 Running 1 363d
kube-system kube-flannel-ds-amd64-gmjth 1/1 Running 1 363d
kube-system kube-flannel-ds-amd64-mvtdg 1/1 Running 1 363d
kube-system kube-proxy-8dtfw 1/1 Running 1 363d
kube-system kube-proxy-9xwgb 1/1 Running 1 363d
kube-system kube-proxy-zcdvn 1/1 Running 1 363d
kube-system kube-scheduler-k8s-master 1/1 Running 1 363d
2.11 更新节点上kubelet证书有效期
$ cp /etc/kubernetes/kubelet.conf{,.default}
#kubeadm init phase kubeconfig kubelet --node-name <节点名称> --kubeconfig-dir /tmp/ --apiserver-advertise-address <集群VIP>，例如：
$ kubeadm init phase kubeconfig kubelet --node-name k8s-master --kubeconfig-dir /tmp/ --apiserver-advertise-address 10.4.7.10
$ \cp /tmp/kubelet.conf /etc/kubernetes/
$ systemctl restart kubelet
kubelet 的配置文件master节点可以和node节点共用！
三、编译源码kubeadm，证书时间自定义
3.1 备份集群配置
$ kubeadm config view > kubeadm-cluster.yaml # 备份
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.18.3", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:15:39Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
# 我这里的版本是1.18.3
3.2 获取对应的kubeadm源码
$ wget https://github.com/kubernetes/kubernetes/archive/v1.18.3.tar.gz
$ tar zxvf v1.18.3.tar.gz
3.3 修改CA证书有效期
$ vim kubernetes-1.18.3/staging/src/k8s.io/client-go/util/cert/cert.go
65 NotBefore: now.UTC(),
66 NotAfter: now.Add(duration365d * 100).UTC(), # 默认是10，改成100
67 KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
68 BasicConstraintsValid: true,
69 IsCA: true,
3.4 修改其他证书有效期
$ vim kubernetes-1.18.3/cmd/kubeadm/app/constants/constants.go
# 跳转至46行，修改如下(追加 * 100):
46 CertificateValidity = time.Hour * 24 * 365 * 100
3.5 安装go环境进行编译
$ wget https://dl.google.com/go/go1.13.9.linux-amd64.tar.gz
$ tar zxf go1.13.9.linux-amd64.tar.gz -C /usr/local/
$ echo 'export PATH=/usr/local/go/bin:$PATH' >> /etc/profile
$ source /etc/profile
$ go version
go version go1.13.9 linux/amd64
3.6 go设置国内代理
Golang V1.13之后支持通过设置变量GOPROXY来修改代理地址，默认的代理服务器，https://proxy.golang.org在国内访问经常出现timeout！详见：https://github.com/goproxy/goproxy.cn/blob/master/README.zh-CN.md
在终端执行即可！
$ go env -w GOPROXY=https://goproxy.cn,direct
$ go env -w GOSUMDB="sum.golang.google.cn"
3.7 编译kubeadm
$ cd kubernetes-1.18.3/ # 进入kubeadm源码目录
$ make all WHAT=cmd/kubeadm GOFLAGS=-v
3.8 替换kubeadm指令
$ cp /usr/bin/kubeadm{,.bak}
$ \cp _output/local/bin/linux/amd64/kubeadm /usr/bin
3.9 更新集群证书
$ kubeadm config view > kubeadm-cluster.yaml
# 如果有多个master节点，请将 kubeadm-cluster.yaml 文件和编译后的kubeadm指令发送至其他master节点
# 更新证书（若有多个master，则需要在所有master上执行）
$ kubeadm alpha certs renew all --config=kubeadm-cluster.yaml
W0904 07:23:15.938694 59308 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
3.10 更新kubeconfig文件
$ rm -f /etc/kubernetes/*.conf
$ kubeadm init phase kubeconfig all --config kubeadm-cluster.yaml
W0904 07:25:41.882636 61426 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
3.11 重启相关pod
在所有Master上执行重启kube-apiserver、kube-controller、kube-scheduler、etcd这4个容器，以便使证书生效。
$ docker ps |egrep "k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd" | awk '{print $1}' | xargs docker restart
3.12 替换admin文件
$ cp ~/.kube/config{,.old}
$ \cp -i /etc/kubernetes/admin.conf ~/.kube/config
$ chown $(id -u):$(id -g) ~/.kube/config
3.13 确认指令正常
$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5b8b769fcd-cpls6 1/1 Running 0 13h
kube-system calico-node-2hk5w 1/1 Running 0 13h
kube-system calico-node-bwmmk 1/1 Running 0 13h
kube-system calico-node-gvldn 1/1 Running 0 13h
kube-system coredns-546565776c-g7j2f 1/1 Running 0 13h
kube-system coredns-546565776c-wtxt4 1/1 Running 0 13h
kube-system etcd-k8s-master 1/1 Running 0 13h
kube-system kube-apiserver-k8s-master 1/1 Running 0 13h
kube-system kube-controller-manager-k8s-master 1/1 Running 1 13h
kube-system kube-proxy-bwkv6 1/1 Running 0 13h
kube-system kube-proxy-jdzps 1/1 Running 0 13h
kube-system kube-proxy-xjpxf 1/1 Running 0 13h
kube-system kube-scheduler-k8s-master 1/1 Running 0 13h
kube-system kuboard-7986796cf8-mk66v 1/1 Running 0 12h
kube-system metrics-server-7f96bbcc66-qldnm 1/1 Running 0 12h
3.14 确认证书更新成功
$ kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Mar 03, 2121 16:02 UTC 2d no
apiserver Mar 03, 2121 16:02 UTC 2d ca no
apiserver-etcd-client Mar 03, 2121 16:02 UTC 2d etcd-ca no
apiserver-kubelet-client Mar 03, 2121 16:02 UTC 2d ca no
controller-manager.conf Mar 03, 2121 16:02 UTC 2d no
etcd-healthcheck-client Mar 03, 2121 16:02 UTC 2d etcd-ca no
etcd-peer Mar 03, 2121 16:02 UTC 2d etcd-ca no
etcd-server Mar 03, 2121 16:02 UTC 2d etcd-ca no
front-proxy-client Mar 03, 2121 16:02 UTC 2d front-proxy-ca no
scheduler.conf Mar 03, 2121 16:02 UTC 2d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Mar 01, 2031 16:02 UTC 9y no
etcd-ca Mar 01, 2031 16:02 UTC 9y no
front-proxy-ca Mar 01, 2031 16:02 UTC 9y no
*************** 当你发现自己的才华撑不起野心时，就请安静下来学习吧！***************
