基于 k8s-v.1.19.7 版本证书更新问题
前言
通过 kubeadm 安装的 k8s 集群，证书有效期为 1年，一年过期后，会导致 api service 不可用，使用过程中会出现报错：
x509: certificate has expired or is not yet valid.
通过实测，记录两种切实可用的方案：
- 手动更新证书（证书已过期也可用）；
- 源码编译kubeadm，使证书时间更长（集群证书已过期也可用）
系统版本
System: CentOS Linux release 7.9.2009 (Core)
kernel: 3.10.0-1160.el7.x86_64
docker版本
Server: 18.06.3-ce
Client: 18.06.3-ce
Kubernetes 版本
v1.19.7
在 master 上使用如下命令查看证书过期时间：
root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
W0818 15:09:56.236357 4122 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 18, 2022 07:04 UTC <invalid> no
apiserver Aug 18, 2022 07:04 UTC <invalid> ca no
apiserver-etcd-client Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
apiserver-kubelet-client Aug 18, 2022 07:04 UTC <invalid> ca no
controller-manager.conf Aug 18, 2022 07:04 UTC <invalid> no
etcd-healthcheck-client Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
etcd-peer Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
etcd-server Aug 18, 2022 07:04 UTC <invalid> etcd-ca no
front-proxy-client Aug 18, 2022 07:04 UTC <invalid> front-proxy-ca no
scheduler.conf Aug 18, 2022 07:04 UTC <invalid> no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 16, 2031 07:04 UTC 8y no
etcd-ca Aug 16, 2031 07:04 UTC 8y no
front-proxy-ca Aug 16, 2031 07:04 UTC 8y no
该命令显示 /etc/kubernetes/pki 文件夹中的客户端证书以及 kubeadm 使用的 KUBECONFIG 文件中嵌入的客户端证书的到期时间/剩余时间。
也可通过如下命令查看证书详细时间：
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | egrep Not
Not Before: Aug 18 07:24:06 2021 GMT
Not After : Aug 18 07:24:06 2022 GMT
执行和 api service的交互报错如下：
root@k8s-master(10.0.0.11)~>kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-08-18T15:10:07+08:00 is after 2022-08-18T07:04:33Z
以上错误表示证书到期，可用如下两种方式进行更新。
手动更新证书
1. 备份
注意：做任何改动之前都需要进行源文件备份
mkdir -pv /etc/kubernetes-bak
cp -r /etc/kubernetes/pki /etc/kubernetes-bak/
cp /etc/kubernetes/*.conf /etc/kubernetes-bak/
etcd 数据目录备份：
cp -ar /var/lib/etcd /var/lib/etcd-bak
2. 执行更新证书
kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
W0818 15:46:40.020407 3775 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
通过上面的命令证书就一键更新完成了，这个时候查看上面的证书可以看到过期时间已经是一年后的时间了：
root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
W0818 15:47:22.712349 3834 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 18, 2023 07:46 UTC 364d no
apiserver Aug 18, 2023 07:46 UTC 364d ca no
apiserver-etcd-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
apiserver-kubelet-client Aug 18, 2023 07:46 UTC 364d ca no
controller-manager.conf Aug 18, 2023 07:46 UTC 364d no
etcd-healthcheck-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-peer Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-server Aug 18, 2023 07:46 UTC 364d etcd-ca no
front-proxy-client Aug 18, 2023 07:46 UTC 364d front-proxy-ca no
scheduler.conf Aug 18, 2023 07:46 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 16, 2031 07:39 UTC 8y no
etcd-ca Aug 16, 2031 07:39 UTC 8y no
front-proxy-ca Aug 16, 2031 07:39 UTC 8y no
3. 更新 kubeconfig 文件
root@k8s-master(10.0.0.11)~>kubeadm init phase kubeconfig all
I0818 15:48:50.617013 3929 version.go:252] remote version is much newer: v1.24.4; falling back to: stable-1.19
W0818 15:48:52.891141 3929 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
4. 覆盖原admin文件
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
5. 重启容器
重启 kube-apiserver、kube-controller、kube-scheduler、etcd 这4个容器即可
docker restart `docker ps | grep etcd | awk '{print $1}'`
docker restart `docker ps | grep kube-apiserver | awk '{print $1}'`
docker restart `docker ps | grep kube-controller | awk '{print $1}'`
docker restart `docker ps | grep kube-scheduler | awk '{print $1}'`
6. 验证是否认证成功
查看pod集群状态，检查刚刚重启的status是否为Running(一般会等待2分钟左右)
kubectl get po -A -o wide
查看当前集群的证书时间，RESIDUAL TIME，为364d，续期一年。
root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 18, 2023 07:46 UTC 364d no
apiserver Aug 18, 2023 07:46 UTC 364d ca no
apiserver-etcd-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
apiserver-kubelet-client Aug 18, 2023 07:46 UTC 364d ca no
controller-manager.conf Aug 18, 2023 07:46 UTC 364d no
etcd-healthcheck-client Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-peer Aug 18, 2023 07:46 UTC 364d etcd-ca no
etcd-server Aug 18, 2023 07:46 UTC 364d etcd-ca no
front-proxy-client Aug 18, 2023 07:46 UTC 364d front-proxy-ca no
scheduler.conf Aug 18, 2023 07:46 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 16, 2031 07:39 UTC 8y no
etcd-ca Aug 16, 2031 07:39 UTC 8y no
front-proxy-ca Aug 16, 2031 07:39 UTC 8y no
到此，通过手动的方式更新证书完成。
源码编译kubeadm
可以在初始化集群之前重新编译 kubeadm，证书有限期可自行指定。
新建集群如下操作：
1. 获取源码
如果集群已经初始化完成，请下载对应版本的源代码进行操作。
我这里的版本为 v1.19.7 去 github 下载对应版本的源代码进行修改编译。
注意：版本一定要一致！
wget https://github.com/kubernetes/kubernetes/archive/v1.19.7.tar.gz
tar -zxvf v1.19.7.tar.gz
mv kubernetes-1.19.7 kubernetes
cd kubernetes
2. 修改CA为100年
>vim ./staging/src/k8s.io/client-go/util/cert/cert.go
...
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
now := time.Now()
tmpl := x509.Certificate{
SerialNumber: new(big.Int).SetInt64(0),
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
NotBefore: now.UTC(),
NotAfter: now.Add(duration365d * 100).UTC(), # 这里10修改为100
...
>vim ./cmd/kubeadm/app/constants/constants.go
...
const (
// KubernetesDir is the directory Kubernetes owns for storing various configuration files
KubernetesDir = "/etc/kubernetes"
// ManifestsSubDirName defines directory name to store manifests
ManifestsSubDirName = "manifests"
// TempDirForKubeadm defines temporary directory for kubeadm
// should be joined with KubernetesDir.
TempDirForKubeadm = "tmp"
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
CertificateValidity = time.Hour * 24 * 365 * 100 # 这里乘以 100 年
...
3. 本地编译
yum install -y gcc make jq rsync
4. 安装go环境
查看 kube-cross 的 TAG 版本号
cat ./build/build-image/cross/VERSION
v1.15.5-1
下载go，上传到 master节点
https://gomirrors.org/dl/go/go1.15.1.linux-amd64.tar.gz
>tar xf go1.15.Linux-amd64.tar.gz -C /usr/local/
# 编辑/etc/profile文件添加如下：
>vim /etc/profile
...
export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$PATH:$GOROOT/bin
>source /etc/profile
验证：
>go version
go version go1.15 linux/amd64
5. 编译kubeadm
# 编译kubeadm, 这里主要编译kubeadm 即可
make all WHAT=cmd/kubeadm GOFLAGS=-v
# 编译kubelet
# make all WHAT=cmd/kubelet GOFLAGS=-v
# 编译kubectl
# make all WHAT=cmd/kubectl GOFLAGS=-v
#编译完产物在 _output/bin/kubeadm 目录下，
#其中bin是使用了软连接
#真实路径是_output/local/bin/Linux/amd64/kubeadm
mv /usr/bin/kubeadm /usr/bin/kubeadm_backup
cp _output/local/bin/Linux/amd64/kubeadm /usr/bin/kubeadm
chmod +x /usr/bin/kubeadm
6. 更新证书
查看证书到期时间：
>kubeadm alpha certs check-expiration
续订全部证书：
kubeadm alpha certs renew all
再次查看证书到期时间：
root@k8s-master(10.0.0.11)~>kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 25, 2121 10:00 UTC 99y no
apiserver Jul 25, 2121 10:00 UTC 99y ca no
apiserver-etcd-client Jul 25, 2121 10:00 UTC 99y etcd-ca no
apiserver-kubelet-client Jul 25, 2121 10:00 UTC 99y ca no
controller-manager.conf Jul 25, 2121 10:00 UTC 99y no
etcd-healthcheck-client Jul 25, 2121 10:00 UTC 99y etcd-ca no
etcd-peer Jul 25, 2121 10:00 UTC 99y etcd-ca no
etcd-server Jul 25, 2121 10:00 UTC 99y etcd-ca no
front-proxy-client Jul 25, 2121 10:00 UTC 99y front-proxy-ca no
scheduler.conf Jul 25, 2121 10:00 UTC 99y no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 25, 2121 10:00 UTC 99y no
etcd-ca Jul 25, 2121 10:00 UTC 99y no
front-proxy-ca Jul 25, 2121 10:00 UTC 99y no
如果集群证书已经过期，在上面的基础继续如下步骤：
docker restart `docker ps | grep etcd | awk '{print $1}'`
docker restart `docker ps | grep kube-apiserver | awk '{print $1}'`
docker restart `docker ps | grep kube-controller | awk '{print $1}'`
docker restart `docker ps | grep kube-scheduler | awk '{print $1}'`
更新 kubeconfig 文件
root@k8s-master(10.0.0.11)~>kubeadm init phase kubeconfig all
I0818 15:48:50.617013 3929 version.go:252] remote version is much newer: v1.24.4; falling back to: stable-1.19
W0818 15:48:52.891141 3929 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
覆盖原admin文件
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
到此，编译更新证书完成。
总结
证书过期的问题需要引起重视，尤其是 1 年期限的证书。 比如外地或国外隔离环境的业务，如果采用默认1年期限的证书，那么就要持续保持1年之内需要去当地维护一次。因此，在建立K8S集群前期就要考虑到证书的问题，建议采用本文中的第二种方式，一劳永逸。
