1. 升级概要
- k8s版本:以 x.y.z 表示,其中 x 是主要版本, y 是次要版本,z 是补丁版本,不能跳过次要版本升级,比如1.28.0->1.30.0,补丁版本可以跳跃更新,比如1.28.0->1.28.10
- 推荐使用与版本匹配的 kubelet 和 kubeadm,最好各组件版本保持一致
- 升级后,因为容器spec的哈希值已更改,所有容器都会被重新启动
- 升级过程需要腾空升每个节点,将工作负载迁移
2. 升级过程
升级工作的基本流程如下:
- 升级主控制平面节点
- 升级其他控制平面节点
- 升级工作节点
2.1 升级主控制平面节点
控制面节点上的升级过程应该每次处理一个节点。 首先选择一个要先行升级的控制面节点。该节点上必须拥有 /etc/kubernetes/admin.conf 文件。本次模拟从1.23.17升级到1.24.15版本,其他版本升级类似。
2.1.1 升级kubeadm
| |
| $ kubectl get nodes |
| NAME STATUS ROLES AGE VERSION |
| k8s-master01 Ready control-plane,master 148d v1.23.17 |
| k8s-node01 Ready <none> 148d v1.23.17 |
| k8s-node02 Ready <none> 148d v1.23.17 |
| k8s-node03 Ready <none> 148d v1.23.17 |
| |
| |
| $ yum list --show-duplicates kubeadm |grep '1.24.' |
| kubeadm.x86_64 1.24.0-0 kubernetes |
| kubeadm.x86_64 1.24.1-0 kubernetes |
| kubeadm.x86_64 1.24.2-0 kubernetes |
| kubeadm.x86_64 1.24.3-0 kubernetes |
| kubeadm.x86_64 1.24.4-0 kubernetes |
| kubeadm.x86_64 1.24.5-0 kubernetes |
| kubeadm.x86_64 1.24.6-0 kubernetes |
| kubeadm.x86_64 1.24.7-0 kubernetes |
| kubeadm.x86_64 1.24.8-0 kubernetes |
| kubeadm.x86_64 1.24.9-0 kubernetes |
| kubeadm.x86_64 1.24.10-0 kubernetes |
| kubeadm.x86_64 1.24.11-0 kubernetes |
| kubeadm.x86_64 1.24.12-0 kubernetes |
| kubeadm.x86_64 1.24.13-0 kubernetes |
| kubeadm.x86_64 1.24.14-0 kubernetes |
| kubeadm.x86_64 1.24.15-0 kubernetes |
| kubeadm.x86_64 1.24.16-0 kubernetes |
| kubeadm.x86_64 1.24.17-0 kubernetes |
| |
| $ yum -y install kubeadm-1.24.15 |
| $ kubeadm version -o yaml |
| clientVersion: |
| buildDate: "2023-06-14T09:54:33Z" |
| compiler: gc |
| gitCommit: 2c67202dc0bb96a7a837cbfb8d72e1f34dfc2808 |
| gitTreeState: clean |
| gitVersion: v1.24.15 |
| goVersion: go1.19.10 |
| major: "1" |
| minor: "24" |
| platform: linux/amd64 |
2.1.2 校验升级计划,不能有报错信息
| $ kubeadm upgrade plan |
| [upgrade/config] Making sure the configuration is correct: |
| [upgrade/config] Reading configuration from the cluster... |
| [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' |
| [preflight] Running pre-flight checks. |
| [upgrade] Running cluster health checks |
| [upgrade] Fetching available versions to upgrade to |
| [upgrade/versions] Cluster version: v1.23.17 |
| [upgrade/versions] kubeadm version: v1.24.15 |
| W0606 18:11:22.084648 122592 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://cdn.dl.k8s.io/release/stable.txt": dial tcp 146.75.113.55:443: i/o timeout (Client.Timeout exceeded while awaiting headers) |
| W0606 18:11:22.084821 122592 version.go:105] falling back to the local client version: v1.24.15 |
| [upgrade/versions] Target version: v1.24.15 |
| [upgrade/versions] Latest version in the v1.23 series: v1.23.17 |
| |
| Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': |
| COMPONENT CURRENT TARGET |
| kubelet 4 x v1.23.17 v1.24.15 |
| |
| Upgrade to the latest stable version: |
| |
| COMPONENT CURRENT TARGET |
| kube-apiserver v1.23.17 v1.24.15 |
| kube-controller-manager v1.23.17 v1.24.15 |
| kube-scheduler v1.23.17 v1.24.15 |
| kube-proxy v1.23.17 v1.24.15 |
| CoreDNS v1.8.6 v1.8.6 |
| etcd 3.5.6-0 3.5.6-0 |
| |
| You can now apply the upgrade by executing the following command: |
| |
| kubeadm upgrade apply v1.24.15 |
| |
| _____________________________________________________________________ |
| |
| |
| The table below shows the current state of component configs as understood by this version of kubeadm. |
| Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or |
| resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually |
| upgrade to is denoted in the "PREFERRED VERSION" column. |
| |
| API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED |
| kubeproxy.config.k8s.io v1alpha1 v1alpha1 no |
| kubelet.config.k8s.io v1beta1 v1beta1 no |
| _____________________________________________________________________ |
2.1.3. 执行升级命令,升级控制面组件
| $ kubeadm upgrade apply v1.24.15 |
| [upgrade/config] Making sure the configuration is correct: |
| [upgrade/config] Reading configuration from the cluster... |
| [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' |
| [preflight] Running pre-flight checks. |
| [upgrade] Running cluster health checks |
| [upgrade/version] You have chosen to change the cluster version to "v1.24.15" |
| [upgrade/versions] Cluster version: v1.23.17 |
| [upgrade/versions] kubeadm version: v1.24.15 |
| [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y |
| [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster |
| [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection |
| [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull' |
| [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.24.15" (timeout: 5m0s)... |
| [upgrade/etcd] Upgrading to TLS for etcd |
| [upgrade/staticpods] Preparing for "etcd" upgrade |
| [upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade |
| [upgrade/etcd] Waiting for etcd to become available |
| [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2379417340" |
| [upgrade/staticpods] Preparing for "kube-apiserver" upgrade |
| [upgrade/staticpods] Renewing apiserver certificate |
| [upgrade/staticpods] Renewing apiserver-kubelet-client certificate |
| [upgrade/staticpods] Renewing front-proxy-client certificate |
| [upgrade/staticpods] Renewing apiserver-etcd-client certificate |
| [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-06-18-17-42/kube-apiserver.yaml" |
| [upgrade/staticpods] Waiting for the kubelet to restart the component |
| [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) |
| [apiclient] Found 1 Pods for label selector component=kube-apiserver |
| [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! |
| [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade |
| [upgrade/staticpods] Renewing controller-manager.conf certificate |
| [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-06-18-17-42/kube-controller-manager.yaml" |
| [upgrade/staticpods] Waiting for the kubelet to restart the component |
| [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) |
| [apiclient] Found 1 Pods for label selector component=kube-controller-manager |
| [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! |
| [upgrade/staticpods] Preparing for "kube-scheduler" upgrade |
| [upgrade/staticpods] Renewing scheduler.conf certificate |
| [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-06-18-17-42/kube-scheduler.yaml" |
| [upgrade/staticpods] Waiting for the kubelet to restart the component |
| [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) |
| [apiclient] Found 1 Pods for label selector component=kube-scheduler |
| [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! |
| [upgrade/postupgrade] Removing the deprecated label node-role.kubernetes.io/master='' from all control plane Nodes. After this step only the label node-role.kubernetes.io/control-plane='' will be present on control plane Nodes. |
| [upgrade/postupgrade] Adding the new taint &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} to all control plane Nodes. After this step both taints &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} and &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} should be present on control plane Nodes. |
| [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace |
| [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster |
| [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" |
| [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes |
| [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials |
| [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token |
| [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster |
| [addons] Applied essential addon: CoreDNS |
| [addons] Applied essential addon: kube-proxy |
| |
| [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.15". Enjoy! |
| |
| [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. |
| |
2.1.4. 腾空该节点,将节点标记为不可调度并驱逐所有负载,准备节点的维护
| $ kubectl drain k8s-master01 --ignore-daemonsets |
| node/k8s-master01 cordoned |
| WARNING: ignoring DaemonSet-managed Pods: default/ds-ng-d6g58, kube-flannel/kube-flannel-ds-qmpmf, kube-system/kube-proxy-tc92t |
| evicting pod kube-system/coredns-65c54cc984-sjd7x |
| pod/coredns-65c54cc984-sjd7x evicted |
| node/k8s-master01 drained |
2.1.5. 升级 kubelet 和 kubectl
| $ yum -y install kubelet-1.24.15 kubectl-1.24.15 |
| $ systemctl daemon-reload |
| $ systemctl restart kubelet |
2.1.6. 解除节点封锁,通过将节点标记为可调度,让其重新上线
| $ kubectl uncordon k8s-master01 |
| node/k8s-master01 uncordoned |
2.2 升级其他控制平面节点
对于其他控制面节点,有两步不一样:
- 不需要执行kubeadm upgrade plan
- kubeadm upgrade node 替换 kubeadm upgrade apply
其他流程一样。
2.3 升级worker工作节点
工作节点上的升级过程应该一次执行一个节点,或者一次执行几个节点, 以不影响运行工作负载所需的最小容量.
2.3.1 升级kubeadm
| $ yum -y install kubeadm-1.24.15 |
| $ kubeadm version -o yaml |
| clientVersion: |
| buildDate: "2023-06-14T09:54:33Z" |
| compiler: gc |
| gitCommit: 2c67202dc0bb96a7a837cbfb8d72e1f34dfc2808 |
| gitTreeState: clean |
| gitVersion: v1.24.15 |
| goVersion: go1.19.10 |
| major: "1" |
| minor: "24" |
| platform: linux/amd64 |
2.3.2 升级kubelet 配置
| $ kubeadm upgrade node |
| [upgrade] Reading configuration from the cluster... |
| [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' |
| [preflight] Running pre-flight checks |
| [preflight] Skipping prepull. Not a control plane node. |
| [upgrade] Skipping phase. Not a control plane node. |
| [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" |
| [upgrade] The configuration for this node was successfully updated! |
| [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager. |
2.3.3 腾空该节点,将节点标记为不可调度并驱逐所有负载,准备节点的维护
| $ kubectl drain k8s-node01 --ignore-daemonsets --force |
| error: unable to drain node "k8s-node01" due to error:cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): default/test, default/web, continuing command... |
| cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-7b584d47bd-kcc7d |
2.3.4 升级 kubelet 和 kubectl
| $ yum -y install kubelet-1.24.15 kubectl-1.24.15 |
| $ systemctl daemon-reload |
| $ systemctl restart kubelet |
2.3.5 解除节点封锁,通过将节点标记为可调度,让其重新上线
| $ kubectl uncordon k8s-node01 |
| node/k8s-node01 uncordoned |
3. 验证集群
查询集群各节点最新版本信息
升级原理:https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
2021-06-06 NGINX配置文件详解
2021-06-06 nginx超时配置