kubeadm部署高可用K8S证书过期

kubeadm部署高可用K8S证书过期

kubeadm 默认证书为一年,一年过期后,会导致 api service 不可用,使用过程中会出现:x509: certificate has expired or is not yet valid.

可以在初始化群集之前重新编译 kubeadm,证书有效期自动为 100 年。

兼容性:

已经测试适用于以下版本:1.17.0、1.18.0、1.19.0、1.20.0、1.21.0、1.22.0、1.23.0

1. 获取源码

访问 https://github.com/kubernetes/kubernetes/releases下载特定版本源码

wget https://github.com/kubernetes/kubernetes/archive/v1.22.2.tar.gz
tar -zxvf kubernetes-1.23.0.tar.gz
mv kubernetes-1.23.0 kubernetes
cd kubernetes

或者使用 git 获取

yum install git
git clone --depth=1 -b v1.22.2 https://github.com/kubernetes/kubernetes.git

2.修改证书有效期

修改证书有效期为100年(默认一年)

vim cmd/kubeadm/app/constants/constants.go
CertificateValidity = time.Hour * 24 * 365 * 99

修改CA有效期为100年(默认十年)

vim ./staging/src/k8s.io/client-go/util/cert/cert.go
const duration365d = time.Hour * 24 * 365

git diff验证,最后修改如下所示,已将证书都修改为100年

[root@master01 kubernetes]# git diff
diff --git a/cmd/kubeadm/app/constants/constants.go b/cmd/kubeadm/app/constants/constants.go
index b355973..c9be686 100644
--- a/cmd/kubeadm/app/constants/constants.go
+++ b/cmd/kubeadm/app/constants/constants.go
@@ -45,7 +45,7 @@ const (
        TempDirForKubeadm = "tmp"
 
        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
-       CertificateValidity = time.Hour * 24 * 365
+       CertificateValidity = time.Hour * 24 * 365 * 99
 
        // CACertAndKeyBaseName defines certificate authority base name
        CACertAndKeyBaseName = "ca"
diff --git a/staging/src/k8s.io/client-go/util/cert/cert.go b/staging/src/k8s.io/client-go/util/cert/cert.go
index bffb152..9e20e3a 100644
--- a/staging/src/k8s.io/client-go/util/cert/cert.go
+++ b/staging/src/k8s.io/client-go/util/cert/cert.go
@@ -64,7 +64,7 @@ func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, erro
                },
                DNSNames:              []string{cfg.CommonName},
                NotBefore:             now.UTC(),
-               NotAfter:              now.Add(duration365d * 10).UTC(),
+               NotAfter:              now.Add(duration365d * 100).UTC(),
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCer
                BasicConstraintsValid: true,
                IsCA:                  true,

3.编译kubeadm

3.1本地环境编译

3.1.1 安装编译软件包

CentOS:

yum install gcc make -y
yum install rsync jq -y

Ubuntu:

sudo apt install build-essential
sudo apt install rsync jq -y

3.1.2 GoLang环境安装

查看kube-cross的TAG版本号

# cat ./build/build-image/cross/VERSION
v1.22.0-go1.16.8-buster.0

#注意:之前的版本显示这种数字格式 v1.16.8-1

安装Go环境

wget https://golang.google.cn/dl/go1.16.8.linux-amd64.tar.gz
tar zxvf go1.16.8.linux-amd64.tar.gz  -C /usr/local

在 /etc/profile 文件添加如下:

#go setting
export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$PATH:$GOROOT/bin

生效

#source /etc/profile

验证:

go version

# 输出如下
go version go1.16.8 linux/amd64

4.编译

#编译 kubeadm, 这里主要编译 kubeadm 即可
make all WHAT=cmd/kubelet GOFLAGS=-v
# 编译 kubectl
make all WHAT=cmd/kubectl GOFLAGS=-v
# 编译 kubelet
make all WHAT=cmd/kubelet GOFLAGS=-v

编译完成后,输出文件位于 _output/bin/kubeadm 目录下

其中 bin 是使用了软连接

真实路径是_output/local/bin/linux/amd64/kubeadm

备份源kubeadm文件

集群模式,所有master节点都要进行kubeadm替换与证书更新

mv /usr/bin/kubeadm /usr/bin/kubeadm_backup

替换kubeadm文件

cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm
chmod +x /usr/bin/kubeadm

查看编译后的版本

kubeadm version

#输出如下
kubeadm version: &version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-dirty", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"dirty", BuildDate:"2022-09-14T01:43:43Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

5.更新证书

如果是使用原版 kubeadm 安装之后,可以手动执行命令更新证书有效期到 100 年。

可以先备份证书,证书在 /etc/kubernetes/pki

5.1 检查证书到期时间

kubeadm certs check-expiration

# 早期版本 (1.19 及之前版本) 命令如下
#kubeadm alpha certs check-expiration

#输出如下
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 21, 2022 08:34 UTC   36d                                     no      
apiserver                  Oct 21, 2022 08:33 UTC   36d             ca                      no      
apiserver-etcd-client      Oct 21, 2022 08:34 UTC   36d             etcd-ca                 no      
apiserver-kubelet-client   Oct 21, 2022 08:34 UTC   36d             ca                      no      
controller-manager.conf    Oct 21, 2022 08:34 UTC   36d                                     no      
etcd-healthcheck-client    Oct 21, 2022 08:34 UTC   36d             etcd-ca                 no      
etcd-peer                  Oct 21, 2022 08:34 UTC   36d             etcd-ca                 no      
etcd-server                Oct 21, 2022 08:34 UTC   36d             etcd-ca                 no      
front-proxy-client         Oct 21, 2022 08:34 UTC   36d             front-proxy-ca          no      
scheduler.conf             Oct 21, 2022 08:34 UTC   36d                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 19, 2031 08:33 UTC   9y              no      
etcd-ca                 Oct 19, 2031 08:34 UTC   9y              no      
front-proxy-ca          Oct 19, 2031 08:34 UTC   9y              no

kubernetes V1.20 开始废弃kubeadm alpha certs 命令。

kubernetes V1.21 开始弃用kubeadm alpha 命令。

5.2 续订所有证书

kubeadm certs renew all

#输出,提示重启控制平面Pod
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates

5.3 再次检查证书有效期

kubeadm certs check-expiration

#输出如下
[root@master01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 22, 2121 01:44 UTC   98y                                     no      
apiserver                  Aug 22, 2121 01:44 UTC   98y             ca                      no      
apiserver-etcd-client      Aug 22, 2121 01:44 UTC   98y             etcd-ca                 no      
apiserver-kubelet-client   Aug 22, 2121 01:44 UTC   98y             ca                      no      
controller-manager.conf    Aug 22, 2121 01:44 UTC   98y                                     no      
etcd-healthcheck-client    Aug 22, 2121 01:44 UTC   98y             etcd-ca                 no      
etcd-peer                  Aug 22, 2121 01:44 UTC   98y             etcd-ca                 no      
etcd-server                Aug 22, 2121 01:44 UTC   98y             etcd-ca                 no      
front-proxy-client         Aug 22, 2121 01:44 UTC   98y             front-proxy-ca          no      
scheduler.conf             Aug 22, 2121 01:44 UTC   98y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Sep 11, 2032 09:09 UTC   9y              no      
etcd-ca                 Sep 11, 2032 09:09 UTC   9y              no      
front-proxy-ca          Sep 11, 2032 09:09 UTC   9y              no 

6.实施后步骤

运行 kubeadm renew 命令后,我们应该重启控制平面Pod。目前并非所有组件和证书都支持动态证书重新加载。由于静态Pod由本地kubelet管理,而不是由API服务器管理,因此无法使用kubectl删除和重新启动它们。要重新启动静态Pod,我们可以暂时从/etc/kubernetes/manifests/中删除其清单文件,然后等待20秒。如果Pod不再位于manifest目录中,kubelet 将终止Pod。然后我们可以将文件移回,在另一个fileCheckFrequency周期之后,kubelet将重新创建 Pod,并且可以完成组件的证书更新。

mkdir -p /opt/temp
mv /etc/kubernetes/manifests/*.yaml /opt/temp

#等待30秒之后
cp /opt/temp/*.yaml /etc/kubernetes/manifests/
posted @   金笛秀才  阅读(920)  评论(0编辑  收藏  举报
相关博文:
阅读排行:
· CSnakes vs Python.NET:高效嵌入与灵活互通的跨语言方案对比
· DeepSeek “源神”启动!「GitHub 热点速览」
· 我与微信审核的“相爱相杀”看个人小程序副业
· Plotly.NET 一个为 .NET 打造的强大开源交互式图表库
· 上周热点回顾(2.17-2.23)
点击右上角即可分享
微信分享提示