1：升级说明
1：可用的K8S集群，使用kubeadm搭建
2：可以小版本升级，也可以跨一个大版本升级，不建议跨两个大版本升级
3：对集群资源做好备份
2：升级目标
现有集群版本已经节点如下：
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 78d v1.23.1
k8s-node1 Ready <none> 78d v1.23.1
k8s-node2 Ready <none> 78d v1.23.1
3：备份集群
kubeadm upgrade 不会影响你的工作负载，只会涉及 Kubernetes 内部的组件，但备份终究是好的。这里主要是对集群的所有资源进行备份，我使用的是一个开源的脚本，项目地址是：https://github.com/solomonxu/k8s-backup-restore
（1）下载脚本
[root@k8s-master ~]# mkdir /data
[root@k8s-master ~]# cd /data/
[root@k8s-master data]# git clone https://github.com/solomonxu/k8s-backup-restore.git
Cloning into 'k8s-backup-restore'...
remote: Enumerating objects: 115, done.
remote: Total 115 (delta 0), reused 0 (delta 0), pack-reused 115
Receiving objects: 100% (115/115), 506.83 KiB | 222.00 KiB/s, done.
Resolving deltas: 100% (38/38), done.
（2）执行备份
[root@k8s-master data]# cd k8s-backup-restore
[root@k8s-master data]# chmod +x ./bin/*
[root@k8s-master data]# ./bin/k8s_backup.sh
4：升级集群
1：Master升级
（1）确定要升级的版本
[root@k8s-master ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes | grep 1.23
kubeadm.x86_64 1.23.1-0 @kubernetes
kubeadm.x86_64 1.23.0-0 kubernetes
kubeadm.x86_64 1.23.1-0 kubernetes
kubeadm.x86_64 1.23.2-0 kubernetes
kubeadm.x86_64 1.23.3-0 kubernetes
kubeadm.x86_64 1.23.4-0 kubernetes
kubeadm.x86_64 1.23.5-0 kubernetes
# 我这里选择1.23.3
（2）升级kubeadm
[root@k8s-master ~]# yum install -y kubeadm-1.23.3-0 --disableexcludes=kubernetes
# 升级完成后验证版本是否正确。
[root@k8s-master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:24:08Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
（3）排空节点
[root@k8s-master ~]# kubectl cordon k8s-master
node/k8s-master cordoned
[root@k8s-master ~]# kubectl drain k8s-master --ignore-daemonsets
node/k8s-master already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-g694p, kube-system/kube-proxy-5vz9w
node/k8s-master drained
（4）运行升级计划，查看是否可以升级
[root@k8s-master ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.23.1
[upgrade/versions] kubeadm version: v1.23.3
[upgrade/versions] Target version: v1.23.5
[upgrade/versions] Latest version in the v1.23 series: v1.23.5
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.23.1 v1.23.5
Upgrade to the latest version in the v1.23 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.23.1 v1.23.5
kube-controller-manager v1.23.1 v1.23.5
kube-scheduler v1.23.1 v1.23.5
kube-proxy v1.23.1 v1.23.5
CoreDNS v1.8.6 v1.8.6
etcd 3.5.1-0 3.5.1-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.23.5
Note: Before you can perform this upgrade, you have to update kubeadm to v1.23.5.
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
上面显示我可以升级到更高版本，不过我这里还是升级到1.23.3。
（5）升级集群
[root@k8s-master ~]# kubeadm upgrade apply v1.23.3
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.23.3"
[upgrade/versions] Cluster version: v1.23.1
[upgrade/versions] kubeadm version: v1.23.3
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.23.3"...
Static pod: kube-apiserver-k8s-master hash: 460a4dd8d578ab992f687a364431cf75
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-master hash: db8e0bd5808dc1ad0e0b24a5aa219a4e
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4180147213"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-04-13-18-16-31/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8s-master hash: 460a4dd8d578ab992f687a364431cf75
Static pod: kube-apiserver-k8s-master hash: 460a4dd8d578ab992f687a364431cf75
Static pod: kube-apiserver-k8s-master hash: 460a4dd8d578ab992f687a364431cf75
Static pod: kube-apiserver-k8s-master hash: 460a4dd8d578ab992f687a364431cf75
Static pod: kube-apiserver-k8s-master hash: f176b87164311a926623cbdbee7acf29
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-04-13-18-16-31/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 5e3dd0c207ce4f0c99822167b7321fca
Static pod: kube-controller-manager-k8s-master hash: 277854bba525837ea292ae3dcdc0ff2e
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-04-13-18-16-31/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: 113d1a662772c436d29bff73d0ad160f
Static pod: kube-scheduler-k8s-master hash: d8b57c48093ffddd1dbe397190de40f3
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.23.3". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
# 由输出可以看出升级执行成功。
（6）取消调度保护
[root@k8s-master ~]# kubectl uncordon k8s-master
node/k8s-master uncordoned
（7）升级节点
[root@k8s-master ~]# kubectl uncordon k8s-master
node/k8s-master uncordoned
[root@k8s-master ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.23.3"...
Static pod: kube-apiserver-k8s-master hash: f176b87164311a926623cbdbee7acf29
Static pod: kube-controller-manager-k8s-master hash: 277854bba525837ea292ae3dcdc0ff2e
Static pod: kube-scheduler-k8s-master hash: d8b57c48093ffddd1dbe397190de40f3
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-master hash: db8e0bd5808dc1ad0e0b24a5aa219a4e
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2944160796"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
（8）升级kubectl和kubelet
[root@k8s-master ~]# yum install -y kubelet-1.23.3-0 kubectl-1.23.3-0 --disableexcludes=kubernetes
...
Updated:
kubectl.x86_64 0:1.23.3-0 kubelet.x86_64 0:1.23.3-0
# 重启kubelet
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart kubelet.service
Node升级
（1）升级kubeadm
[root@k8s-node1 ~]# yum install -y kubeadm-1.23.3-0 --disableexcludes=kubernetes
Updated:
kubeadm.x86_64 0:1.23.3-0
（2）设置节点不可调度并排空节点
[root@k8s-master ~]# kubectl drain k8s-node1 --ignore-daemonsets
node/k8s-node1 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-9rvvx, kube-system/kube-proxy-8gptv
evicting pod ingress-nginx/ingress-nginx-controller-7fc8d55869-lmjjc
evicting pod default/nginx-9fbb7d78-g84zn
evicting pod ingress-nginx/ingress-nginx-admission-create-vwfjw
pod/ingress-nginx-admission-create-vwfjw evicted
pod/nginx-9fbb7d78-g84zn evicted
pod/ingress-nginx-controller-7fc8d55869-lmjjc evicted
node/k8s-node1 drained
（3）升级节点
[root@k8s-node1 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
（4）升级kubelet
[root@k8s-node1 ~]# yum install -y kubelet-1.23.3-0 --disableexcludes=kubernetes
---
Updated:
kubelet.x86_64 0:1.23.3-0
重启kubelet
[root@k8s-node1 ~]# systemctl daemon-reload
[root@k8s-node1 ~]# systemctl restart kubelet
（5）设置节点可调度
[root@k8s-master ~]# kubectl uncordon k8s-node1
node/k8s-node1 uncordoned
# node2同样操作
5：验证集群
（1）验证集群状态是否正常
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 78d v1.23.3
k8s-node1 Ready <none> 78d v1.23.3
k8s-node2 Ready <none> 78d v1.23.3
（2）验证集群证书是否正常
[root@k8s-master ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Apr 13, 2023 22:19 UTC 364d ca no
apiserver Apr 13, 2023 22:16 UTC 364d ca no
apiserver-etcd-client Apr 13, 2023 22:16 UTC 364d etcd-ca no
apiserver-kubelet-client Apr 13, 2023 22:16 UTC 364d ca no
controller-manager.conf Apr 13, 2023 22:17 UTC 364d ca no
etcd-healthcheck-client Mar 20, 2122 21:29 UTC 99y etcd-ca no
etcd-peer Mar 20, 2122 21:29 UTC 99y etcd-ca no
etcd-server Mar 20, 2122 21:29 UTC 99y etcd-ca no
front-proxy-client Apr 13, 2023 22:16 UTC 364d front-proxy-ca no
scheduler.conf Apr 13, 2023 22:17 UTC 364d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jan 23, 2032 03:04 UTC 9y no
etcd-ca Jan 23, 2032 03:04 UTC 9y no
front-proxy-ca Jan 23, 2032 03:04 UTC 9y no
# 这里出现 2122是因为我做了100年的证书所以才会出现的
[root@k8s-master ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default httpd-6496d888c9-fm9zv 1/1 Running 0 9m46s
default nginx-9fbb7d78-bcvz6 1/1 Running 0 9m46s
ingress-nginx ingress-nginx-controller-7fc8d55869-qgbx8 1/1 Running 0 9m46s
kube-system calico-kube-controllers-85b5b5888d-sv8z7 1/1 Running 0 9m46s
kube-system calico-node-8q54t 1/1 Running 2 (35h ago) 78d
kube-system calico-node-9rvvx 1/1 Running 1 (23d ago) 78d
kube-system calico-node-g694p 1/1 Running 1 (23d ago) 78d
kube-system coredns-6d8c4cb4d-8nm8n 1/1 Running 0 9m46s
kube-system coredns-6d8c4cb4d-lqtn7 1/1 Running 0 9m46s
kube-system etcd-k8s-master 1/1 Running 2 (70m ago) 78d
kube-system kube-apiserver-k8s-master 1/1 Running 0 31m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 31m
kube-system kube-proxy-6wjss 1/1 Running 0 30m
kube-system kube-proxy-8gptv 1/1 Running 0 30m
kube-system kube-proxy-d6fvw 1/1 Running 0 29m
kube-system kube-scheduler-k8s-master 1/1 Running 0 30m
注意：kubeadm upgrade 也会自动对它在此节点上管理的证书进行续约。 如果选择不对证书进行续约，可以使用 --certificate-renewal=false。
故障恢复
在升级过程中如果升级失败并且没有回滚，可以继续执行kubeadm upgrade。如果要从故障状态恢复，可以执行kubeadm upgrade --force。
在升级期间，会在/etc/kubernetes/tmp目录下生成备份文件：
1：kubeadm-backup-etcd-
2：kubeadm-backup-manifests-
kubeadm-backup-etcd中包含本地etcd的数据备份，如果升级失败并且无法修复，可以将其数据复制到etcd数据目录进行手动修复。
kubeadm-backup-manifests中保存的是节点静态pod的YAML清单，如果升级失败并且无法修复，可以将其复制到/etc/kubernetes/manifests下进行手动修复。
