解决kubeadm部署的k8s 集群证书过期

目录

 

K8S CA证书是10年，但是组件证书的日期只有1年，为了证书一直可用状态需要更新，目前主流的一共有3种：

1、版本升级，只要升级就会让各个证书延期1年，官方设置1年有效期的目的就是希望用户在一年内能升级1次；
2、通过命令续期 （这种只能延长一年）；
3、编译源码Kubeadm，证书有效期可自定义；

本实验环境是单master集群环境，如果是多master集群环境那么需要将master上更新的证书分发到各个节点上！

此文档采用K8s 1.18.3版本，不保证其他版本也适用，建议自行测试。

一、查看证书过期时间

1.1 方式一

  $ kubeadm alpha certs check-expiration

个人搭建的集群不知道为什么，使用上述命令无法查看ca的证书有效期，所以记录上方式二！

1.2 方式二

  $ for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;
  do openssl x509 -in $item -text -noout| grep Not;
  echo ======================$item===============;
  done

也可以一个一个的进行查看：

  $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '

二、通过命令续期

2.1 修改集群内所有机器的时间，模拟证书在过期的边缘

  $ date -s "2022-3-1 12:00"

2.2 查看证书有效期

为了更直观的看到证书的有效期！

  $ kubeadm alpha certs check-expiration
  [check-expiration] Reading configuration from the cluster...
  [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
   
  CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
  admin.conf Mar 03, 2022 16:02 UTC 2d no
  apiserver Mar 03, 2022 16:02 UTC 2d ca no
  apiserver-etcd-client Mar 03, 2022 16:02 UTC 2d etcd-ca no
  apiserver-kubelet-client Mar 03, 2022 16:02 UTC 2d ca no
  controller-manager.conf Mar 03, 2022 16:02 UTC 2d no
  etcd-healthcheck-client Mar 03, 2022 16:02 UTC 2d etcd-ca no
  etcd-peer Mar 03, 2022 16:02 UTC 2d etcd-ca no
  etcd-server Mar 03, 2022 16:02 UTC 2d etcd-ca no
  front-proxy-client Mar 03, 2022 16:02 UTC 2d front-proxy-ca no
  scheduler.conf Mar 03, 2022 16:02 UTC 2d no
   
  CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
  ca Mar 01, 2031 16:02 UTC 9y no
  etcd-ca Mar 01, 2031 16:02 UTC 9y no
  front-proxy-ca Mar 01, 2031 16:02 UTC 9y no

如果证书过期的话，就会出现以下情况：

  $ kubectl get pod -n kube-system
  Unable to connect to the server: x509: certificate has expired or is not yet valid

2.3 备份原有数据

  $ kubeadm config view > /root/kubeadm.yaml
  $ cat /root/kubeadm.yaml
  apiServer:
  extraArgs:
  authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta2
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  controllerManager: {}
  dns:
  type: CoreDNS
  etcd:
  local:
  dataDir: /var/lib/etcd
  imageRepository: registry.aliyuncs.com/google_containers
  kind: ClusterConfiguration
  kubernetesVersion: v1.18.3
  networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
  scheduler: {}

2.4 备份证书

备份主要就是为了升级失败之后，便于回滚！

  $ cp -rp /etc/kubernetes /etc/kubernetes_$(date +%F)
  $ ls /etc/kubernetes_2022-03-01/
  admin.conf controller-manager.conf kubelet.conf manifests pki scheduler.conf

2.5 更新证书

  $ kubeadm alpha certs renew all --config=/root/kubeadm.yaml

2.6 确认证书有效期

  $ kubeadm alpha certs check-expiration
  CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
  admin.conf Mar 01, 2023 04:02 UTC 364d no
  apiserver Mar 01, 2023 04:02 UTC 364d no
  apiserver-etcd-client Mar 01, 2023 04:02 UTC 364d no
  apiserver-kubelet-client Mar 01, 2023 04:02 UTC 364d no
  controller-manager.conf Mar 01, 2023 04:02 UTC 364d no
  etcd-healthcheck-client Mar 01, 2023 04:02 UTC 364d no
  etcd-peer Mar 01, 2023 04:02 UTC 364d no
  etcd-server Mar 01, 2023 04:02 UTC 364d no
  front-proxy-client Mar 01, 2023 04:02 UTC 364d no
  scheduler.conf Mar 01, 2023 04:02 UTC 364d no

2.7 更新kubeconfig文件

  $ rm -f /etc/kubernetes/*.conf
  $ kubeadm init phase kubeconfig all --config /root/kubeadm.yaml

2.8 更新客户端证书

  $ cp $HOME/.kube/config{,.default}
  $ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  $ chown $(id -u):$(id -g) $HOME/.kube/config

2.9 重启相关的pod

  $ docker ps |egrep "k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd" | awk '{print $1}' | xargs docker rm -f

也可以干脆一点，直接重启docker即可！

2.10 查看pod是否运行正常

  $ kubectl get pod -A
  NAMESPACE NAME READY STATUS RESTARTS AGE
  kube-system coredns-58cc8c89f4-8lq2k 1/1 Running 1 363d
  kube-system coredns-58cc8c89f4-hz774 1/1 Running 1 363d
  kube-system etcd-k8s-master 1/1 Running 1 363d
  kube-system kube-apiserver-k8s-master 1/1 Running 1 363d
  kube-system kube-controller-manager-k8s-master 1/1 Running 1 363d
  kube-system kube-flannel-ds-amd64-fh9nx 1/1 Running 1 363d
  kube-system kube-flannel-ds-amd64-gmjth 1/1 Running 1 363d
  kube-system kube-flannel-ds-amd64-mvtdg 1/1 Running 1 363d
  kube-system kube-proxy-8dtfw 1/1 Running 1 363d
  kube-system kube-proxy-9xwgb 1/1 Running 1 363d
  kube-system kube-proxy-zcdvn 1/1 Running 1 363d
  kube-system kube-scheduler-k8s-master 1/1 Running 1 363d

2.11 更新节点上kubelet证书有效期

  $ cp /etc/kubernetes/kubelet.conf{,.default}
  #kubeadm init phase kubeconfig kubelet --node-name <节点名称> --kubeconfig-dir /tmp/ --apiserver-advertise-address <集群VIP>，例如：
  $ kubeadm init phase kubeconfig kubelet --node-name k8s-master --kubeconfig-dir /tmp/ --apiserver-advertise-address 10.4.7.10
  $ \cp /tmp/kubelet.conf /etc/kubernetes/
  $ systemctl restart kubelet

kubelet 的配置文件master节点可以和node节点共用！

三、编译源码kubeadm，证书时间自定义

3.1 备份集群配置

  $ kubeadm config view > kubeadm-cluster.yaml # 备份
  $ kubeadm version
  kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.18.3", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:15:39Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
  # 我这里的版本是1.18.3

3.2 获取对应的kubeadm源码

  $ wget https://github.com/kubernetes/kubernetes/archive/v1.18.3.tar.gz
  $ tar zxvf v1.18.3.tar.gz

3.3 修改CA证书有效期

  $ vim kubernetes-1.18.3/staging/src/k8s.io/client-go/util/cert/cert.go
  65 NotBefore: now.UTC(),
  66 NotAfter: now.Add(duration365d * 100).UTC(), # 默认是10，改成100
  67 KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
  68 BasicConstraintsValid: true,
  69 IsCA: true,

3.4 修改其他证书有效期

  $ vim kubernetes-1.18.3/cmd/kubeadm/app/constants/constants.go
  # 跳转至46行，修改如下(追加 * 100):
  46 CertificateValidity = time.Hour * 24 * 365 * 100

3.5 安装go环境进行编译

  $ wget https://dl.google.com/go/go1.13.9.linux-amd64.tar.gz
  $ tar zxf go1.13.9.linux-amd64.tar.gz -C /usr/local/
  $ echo 'export PATH=/usr/local/go/bin:$PATH' >> /etc/profile
  $ source /etc/profile
  $ go version
  go version go1.13.9 linux/amd64

3.6 go设置国内代理

Golang V1.13之后支持通过设置变量GOPROXY来修改代理地址，默认的代理服务器，https://proxy.golang.org在国内访问经常出现timeout！详见：https://github.com/goproxy/goproxy.cn/blob/master/README.zh-CN.md
在终端执行即可！

  $ go env -w GOPROXY=https://goproxy.cn,direct
  $ go env -w GOSUMDB="sum.golang.google.cn"

3.7 编译kubeadm

  $ cd kubernetes-1.18.3/ # 进入kubeadm源码目录
  $ make all WHAT=cmd/kubeadm GOFLAGS=-v

3.8 替换kubeadm指令

  $ cp /usr/bin/kubeadm{,.bak}
  $ \cp _output/local/bin/linux/amd64/kubeadm /usr/bin

3.9 更新集群证书

  $ kubeadm config view > kubeadm-cluster.yaml
  # 如果有多个master节点，请将 kubeadm-cluster.yaml 文件和编译后的kubeadm指令发送至其他master节点
   
  # 更新证书（若有多个master，则需要在所有master上执行）
  $ kubeadm alpha certs renew all --config=kubeadm-cluster.yaml
  W0904 07:23:15.938694 59308 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
  certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
  certificate for serving the Kubernetes API renewed
  certificate the apiserver uses to access etcd renewed
  certificate for the API server to connect to kubelet renewed
  certificate embedded in the kubeconfig file for the controller manager to use renewed
  certificate for liveness probes to healthcheck etcd renewed
  certificate for etcd nodes to communicate with each other renewed
  certificate for serving etcd renewed
  certificate for the front proxy client renewed
  certificate embedded in the kubeconfig file for the scheduler manager to use renewed

3.10 更新kubeconfig文件

  $ rm -f /etc/kubernetes/*.conf
  $ kubeadm init phase kubeconfig all --config kubeadm-cluster.yaml
  W0904 07:25:41.882636 61426 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
  [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
  [kubeconfig] Writing "admin.conf" kubeconfig file
  [kubeconfig] Writing "kubelet.conf" kubeconfig file
  [kubeconfig] Writing "controller-manager.conf" kubeconfig file
  [kubeconfig] Writing "scheduler.conf" kubeconfig file

3.11 重启相关pod

在所有Master上执行重启kube-apiserver、kube-controller、kube-scheduler、etcd这4个容器，以便使证书生效。

  $ docker ps |egrep "k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd" | awk '{print $1}' | xargs docker restart

3.12 替换admin文件

  $ cp ~/.kube/config{,.old}
  $ \cp -i /etc/kubernetes/admin.conf ~/.kube/config
  $ chown $(id -u):$(id -g) ~/.kube/config

3.13 确认指令正常

  $ kubectl get pod -A
  NAMESPACE NAME READY STATUS RESTARTS AGE
  kube-system calico-kube-controllers-5b8b769fcd-cpls6 1/1 Running 0 13h
  kube-system calico-node-2hk5w 1/1 Running 0 13h
  kube-system calico-node-bwmmk 1/1 Running 0 13h
  kube-system calico-node-gvldn 1/1 Running 0 13h
  kube-system coredns-546565776c-g7j2f 1/1 Running 0 13h
  kube-system coredns-546565776c-wtxt4 1/1 Running 0 13h
  kube-system etcd-k8s-master 1/1 Running 0 13h
  kube-system kube-apiserver-k8s-master 1/1 Running 0 13h
  kube-system kube-controller-manager-k8s-master 1/1 Running 1 13h
  kube-system kube-proxy-bwkv6 1/1 Running 0 13h
  kube-system kube-proxy-jdzps 1/1 Running 0 13h
  kube-system kube-proxy-xjpxf 1/1 Running 0 13h
  kube-system kube-scheduler-k8s-master 1/1 Running 0 13h
  kube-system kuboard-7986796cf8-mk66v 1/1 Running 0 12h
  kube-system metrics-server-7f96bbcc66-qldnm 1/1 Running 0 12h

3.14 确认证书更新成功

  $ kubeadm alpha certs check-expiration
  [check-expiration] Reading configuration from the cluster...
  [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
   
  CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
  admin.conf Mar 03, 2121 16:02 UTC 2d no
  apiserver Mar 03, 2121 16:02 UTC 2d ca no
  apiserver-etcd-client Mar 03, 2121 16:02 UTC 2d etcd-ca no
  apiserver-kubelet-client Mar 03, 2121 16:02 UTC 2d ca no
  controller-manager.conf Mar 03, 2121 16:02 UTC 2d no
  etcd-healthcheck-client Mar 03, 2121 16:02 UTC 2d etcd-ca no
  etcd-peer Mar 03, 2121 16:02 UTC 2d etcd-ca no
  etcd-server Mar 03, 2121 16:02 UTC 2d etcd-ca no
  front-proxy-client Mar 03, 2121 16:02 UTC 2d front-proxy-ca no
  scheduler.conf Mar 03, 2121 16:02 UTC 2d no
   
  CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
  ca Mar 01, 2031 16:02 UTC 9y no
  etcd-ca Mar 01, 2031 16:02 UTC 9y no
  front-proxy-ca Mar 01, 2031 16:02 UTC 9y no
*************** 当你发现自己的才华撑不起野心时，就请安静下来学习吧！***************
posted @ 2024-02-08 11:04  94xiaoyu  阅读(1788)  评论(0)  编辑  收藏  举报