kubeadm初始化集群
k8s 安装方式
安装方式 | 使用环境 | |
---|---|---|
kubeadm k8s | k8s 官方支持 | |
kops | 亚马逊公有云的工具 | |
kubespray | 最早是基于ansiable安装 | |
kubeasz | 基于ansible而二进制安装 | 我接触到的公司在用 |
kubespray | 基于ansible而二进制安装 | 有公司再用 |
kubekey | 基于ansible而二进制安装 | 青云公司维护 |
k8s发行版 rancher 、openshift
基础环境准备
ubuntu发行版本
root@testwedcom:~# cat /etc/issue
Ubuntu 22.04.1 LTS \n \l
第一步:修改网卡设置
root@testwedcom:~# cat /etc/netplan/00-installer-config.yaml
network:
version: 2
ethernets:
ens33:
dhcp4: false
addresses: [10.4.7.100/24,]
#gateway4: 10.4.7.254 提示已弃用,被routes替代
routes:
- to: default
via: 10.4.7.254
metric: 100
on-link: true
nameservers:
addresses: [114.114.114.114,]
修改后生效
netplan apply
第二步:修改主机名
hostnamectl set-hostname master01
第三步:设置时间
timedatectl set-timezone Asia/Shanghai
第四步:关闭swap
swapoff -a
找到/etc/fstab
注释掉包含swap的行
第五步:关闭SELINUX/防火墙
apt install policycoreutils-python-utils -y
setenforce 0
ufw disable
第六步:离线安装.deb
v1.24 之前的 Kubernetes 版本直接集成了 Docker Engine 的一个组件,名为 dockershim。之后的版本需要安装cri-dockerd+docker-ce
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.6/cri-dockerd_0.2.6.3-0.ubuntu-jammy_amd64.deb -O cri-dockerd_0.2.6.3-0.ubuntu-jammy_amd64.deb &&\
apt-get install ./cri-dockerd_0.2.6.3-0.ubuntu-jammy_amd64.deb -y &&\
systemctl status cri-docker
-
安装docker
wget https://get.docker.com/ -O getdocker.sh;sh ./getdocker.sh --mirror Aliyun &&\ docker info
-
安装kubelet、kubeadm、kubectl
apt-get update && apt-get install -y apt-transport-https curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
第七步:内核规则
k8s中使用到了linux的虚拟网桥,虚拟网桥属于二层通信,bridge-nf-call-iptables=1 表示在二层数据转发时也会调用netfilter中的三层规则(conntrack),进行连接跟踪
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 >/proc/sys/net/bridge/bridge-nf-call-ip6tables
echo 1 >/proc/sys/net/ipv4/ip_forward
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
第八步:初始化前的准备工作
准备镜像
由于无法拉取google仓库的镜像,因此使用dockerhub上的镜像
#root@master01:~# kubeadm config images list
registry.k8s.io/kube-apiserver:v1.25.3
registry.k8s.io/kube-controller-manager:v1.25.3
registry.k8s.io/kube-scheduler:v1.25.3
registry.k8s.io/kube-proxy:v1.25.3
registry.k8s.io/pause:3.8
registry.k8s.io/etcd:3.5.4-0
registry.k8s.io/coredns/coredns:v1.9.3
docker pull k8simage/kube-apiserver:v1.25.3
docker pull k8simage/kube-controller-manager:v1.25.3
docker pull k8simage/kube-scheduler:v1.25.3
docker pull k8simage/kube-proxy:v1.25.3
docker pull k8simage/pause:3.8
docker pull k8simage/etcd:3.5.4-0
docker pull k8simage/coredns:v1.9.3
docker tag k8simage/kube-apiserver:v1.25.3 registry.k8s.io/kube-apiserver:v1.25.3
docker tag k8simage/kube-controller-manager:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3
docker tag k8simage/kube-scheduler:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3
docker tag k8simage/kube-proxy:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3
docker tag k8simage/pause:3.8 registry.k8s.io/pause:3.8
docker tag k8simage/etcd:3.5.4-0 registry.k8s.io/etcd:3.5.4-0
docker tag k8simage/coredns:v1.9.3 registry.k8s.io/coredns/coredns:v1.9.3
docker images | awk '/^k8s/{print $1":"$2}'|xargs docker rmi {}
修改cri-docker的infra镜像为kubeadm config images list
的镜像
root@test:~# vi /lib/systemd/system/cri-docker.service
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image registry.k8s.io/pause:3.8 --container-runtime-endpoint fd://
root@test:~# systemctl daemon-reload
root@test:~# systemctl restart cri-docker
单控制平面
第九步:初始化master
kubeadm init --pod-network-cidr="172.7.0.0/16" --service-cidr="192.168.0.0/16" --cri-socket=unix:///var/run/cri-dockerd.sock
#kubeadm reset --cri-socket unix:///var/run/cri-dockerd.sock -v 5
拷贝kubeconfig文件到默认路径
mkdir -p $HOME/.kube
chown $(id -u):$(id -g) $HOME/.kube/config
此时coredns为pending状态,需要安装cni插件。待所有组件正常后可以执行后续节点的加入动作
kubectl get pod -A
curl -sSl https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml -o flannel.yaml
可选项部署网络策略插件
curl -sfL https://docs.projectcalico.org/manifests/canal.yaml | kubectl apply -f -
calico插件地址 https://docs.tigera.io/archive/v3.15/getting-started/kubernetes/quickstart
calicoctl地址 https://github.com/projectcalico/calico/releases/download/v3.15.0/release-v3.15.0.tgz
第十步:加入node
kubeadm join 10.4.7.100:6443 --token 19s9ky.bi78046dbcqgttje \
--discovery-token-ca-cert-hash sha256:2cfa40e6f148ed3e9af203dad1a894ba4a2cfdad19aaea7a763d11064f917dbe \
--cri-socket unix:///var/run/cri-dockerd.sock
至此coredns的2个副本可能运行在同一个节点上,为了实现高可用需要重新调度
kubectl rollout restart deploy coredns -n kube-system
高可用集群部署
第一步:创建kube-apiserver的LB
ip a a 10.4.7.10 dev ens33
ipvsadm -A -t 10.4.7.10:16443 -s rr
ipvsadm -a -t 10.4.7.10:16443 -r 10.4.7.100:6443 -m -w 1
ipvsadm -a -t 10.4.7.10:16443 -r 10.4.7.101:6443 -m -w 1
ipvsadm -a -t 10.4.7.10:16443 -r 10.4.7.102:6443 -m -w 1
ipvsadm -Ln
第二步:初始化控制平面
# --upload-certs 将控制平面证书上传到 kubeadm-certs Secret
# kubeadm reset --cri-socket unix:///var/run/cri-dockerd.sock -v 5
kubeadm init --control-plane-endpoint "10.4.7.10:16443" \
--upload-certs \
--pod-network-cidr="172.7.0.0/16" \
--service-cidr="192.168.0.0/16" \
--cri-socket unix:///var/run/cri-dockerd.sock
拷贝kubeconfig文件到默认路径
mkdir -p $HOME/.kube
chown $(id -u):$(id -g) $HOME/.kube/config
此时coredns为pending状态,需要安装cni插件。待所有组件正常后可以执行后续节点的加入动作
kubectl get pod -A
curl -sSl https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml -o flannel.yaml
可选项部署网络策略插件
curl -sfL https://docs.projectcalico.org/manifests/canal.yaml | kubectl apply -f -
第三步:加入master
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
#You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 10.4.7.10:16443 --token bk3kic.dfnm50tve8qjvxk6 \
--discovery-token-ca-cert-hash sha256:85d4b8384118a07e3149a8ffeaa5981b8a1888762d52fa0f2d710ff0ad95b84b \
--control-plane --certificate-key 5c95e3d868cf41531a57b9d213ee880fae41004d0e96183e04a8a974da0122f8 \
--cri-socket unix:///var/run/cri-dockerd.sock
第四步:加入node
# Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.4.7.10:16443 --token bk3kic.dfnm50tve8qjvxk6 \
--discovery-token-ca-cert-hash sha256:85d4b8384118a07e3149a8ffeaa5981b8a1888762d52fa0f2d710ff0ad95b84b \
--cri-socket unix:///var/run/cri-dockerd.sock
至此coredns的2个副本可能运行在同一个节点上,为了实现高可用需要重新调度
kubectl rollout restart deploy coredns -n kube-system
第五步:测试
kubectl run mypod --image=alpine --restart=Never -- sleep 100
kubectl create svc clusterip mypod --tcp=80
kubectl label pod mypod app=mypod
集群升级
对上面的高可用集群进行版本升级/降级。从1.25.3 降级到1.25.0
首先升级master
第一步:下线节点禁止调度任务
kubectl cordon master01
kubectl drain master01 --ignore-daemonsets
第二步:升级/降级kubeadm
准备镜像
docker pull k8simage/kube-apiserver:v1.25.0
docker pull k8simage/kube-controller-manager:v1.25.0
docker pull k8simage/kube-scheduler:v1.25.0
docker pull k8simage/kube-proxy:v1.25.0
docker tag k8simage/kube-apiserver:v1.25.0 registry.k8s.io/kube-apiserver:v1.25.0
docker tag k8simage/kube-controller-manager:v1.25.0 registry.k8s.io/kube-controller-manager:v1.25.0
docker tag k8simage/kube-scheduler:v1.25.0 registry.k8s.io/kube-scheduler:v1.25.0
docker tag k8simage/kube-proxy:v1.25.0 registry.k8s.io/kube-proxy:v1.25.0
# 查看有哪些可用版本
#yum list kubelet --showduplicates
# kubelet-版本号
#yum install kubelet-1.24.0-0 kubeadm-1.24.0-0 -y
#systemctl enable kubelet.service --now
apt-cache madison kubeadm
apt-get update && apt-get install -y kubeadm=1.25.0-00 --allow-downgrades
#check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter
kubeadm upgrade plan
kubeadm upgrade apply v1.25.0 --etcd-upgrade=true
升级成功后提示
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.25.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
第三步:升级kubelet/kubectl
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.25.0-00 kubectl=1.25.0-00 --allow-downgrades && \
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
第四步:验证
kubectl uncordon master01
kubectl get nodes
升级其他master节点
# 其他不变,唯一需要修改第二步中kubeadm update apply v1.25.0 为kubeadm upgrade node
kubeadm upgrade node --etcd-upgrade=true
升级worker节点
第一步:下线节点禁止调度任务
kubectl cordon node01
kubectl drain node01 --ignore-daemonsets
第二步:升级/降级kubeadm
kubeadm upgrade node
第三步:升级kubelet/kubectl
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.25.0-00 kubectl=1.25.0-00 --allow-downgrades && \
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
第四步:验证
kubectl uncordon node01
kubectl get nodes
异常恢复
要从故障状态恢复,你还可以运行 kubeadm upgrade apply --force
而无需更改集群正在运行的版本。
在升级期间,kubeadm 向 /etc/kubernetes/tmp
目录下的如下备份文件夹写入数据:
kubeadm-backup-etcd-<date>-<time>
kubeadm-backup-manifests-<date>-<time>
kubeadm-backup-etcd
包含当前控制面节点本地 etcd 成员数据的备份。 如果 etcd 升级失败并且自动回滚也无法修复,则可以将此文件夹中的内容复制到 /var/lib/etcd
进行手工修复。如果使用的是外部的 etcd,则此备份文件夹为空。
kubeadm-backup-manifests
包含当前控制面节点的静态 Pod 清单文件的备份版本。 如果升级失败并且无法自动回滚,则此文件夹中的内容可以复制到 /etc/kubernetes/manifests
目录实现手工恢复。 如果由于某些原因,在升级前后某个组件的清单未发生变化,则 kubeadm 也不会为之 生成备份版本。
etcd备份和恢复
备份到/data/backup.db
kubectl exec etcd-master01 -n kube-system -- \
etcdctl --endpoints=10.4.7.100:2379 \
--cacert="/etc/kubernetes/pki/etcd/ca.crt" \
--cert="/etc/kubernetes/pki/etcd/server.crt" \
--key="/etc/kubernetes/pki/etcd/server.key" \
snapshot save /data/backup.db
恢复/data/backup.db
kubectl exec etcd-master01 -n kube-system -- \
etcdctl --endpoints=10.4.7.100:2379 \
--cacert="/etc/kubernetes/pki/etcd/ca.crt" \
--cert="/etc/kubernetes/pki/etcd/server.crt" \
--key="/etc/kubernetes/pki/etcd/server.key" \
snapshot restore /data/backup.db
节点下线
kubectl drain <node name> --delete-emptydir-data --force --ignore-daemonsets
kubeadm reset
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm -C
kubectl delete node <node name
加入节点
#上传证书到secret
root@master01:/etc/kubernetes/manifests# kubeadm init phase upload-certs --upload-certs --config /tmp/1.yaml
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
bf0ea78ad23fffbe4d60bc5818f47e724d26ad85873d4a412eb8f18f19e36b49
#生成ca的hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
85d4b8384118a07e3149a8ffeaa5981b8a1888762d52fa0f2d710ff0ad95b84b
#查看token
kubeadm token list
#kubeadm token creat
master加入集群
kubeadm join 192.168.122.10:6443 --token ay2qz0.yybdusqgv3i7tvzm \
--discovery-token-ca-cert-hash sha256:333cfe7e6cbdec5b7c50a042c7f12b7150ea6a4d4b6522b27f1192af4c32150f \
--control-plane --certificate-key bf0ea78ad23fffbe4d60bc5818f47e724d26ad85873d4a412eb8f18f19e36b49 \
--cri-socket unix:///var/run/cri-dockerd.sock
node 加入集群
kubeadm join 10.4.7.100:6443 --token ay2qz0.yybdusqgv3i7tvzm \
--discovery-token-ca-cert-hash sha256:333cfe7e6cbdec5b7c50a042c7f12b7150ea6a4d4b6522b27f1192af4c32150f \
--cri-socket unix:///var/run/cri-dockerd.sock
证书轮替
使用kubeadm 安装的集群默认ca 10年,证书为1年。证书对k8s集权至关重要,一旦证书过期k8s 各组件之间通信将无法进行,在v1.25中已经可以使用以下命令操作
一旦发现证书即将过期,需要在所有控制平面执行以下命令
第一步:更新证书
#除了all 之外也可单独更新单个组件的证书
kubeadm certs renew all
查看更新结果
kubeadm certs check-expiration
第二步:重启服务
You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
dashboard
第一步:部署
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
# 修改 kubectl edit kubernetes-dashboard -n kubernetes-dashboard 为nodeport 类型
第二步:鉴权
kubectl create sa dashboard
cat <<EOF|kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: secret-dashboard
annotations:
kubernetes.io/service-account.name: dashboard # 与sa名称保持一致
type: kubernetes.io/service-account-token
EOF
kubectl create clusterrolebinding dashboard --clusterrole=cluster-admin --serviceaccount=default:dashboard
kubectl describe secret secret-dashboard|awk '/token/{print $2}'
第三步:清理环境
kubectl delete clusterrolebinding dashboard
kubectl delete sa dashboard
metrics-server
Metrics Server是 Kubernetes 容器资源指标的可扩展、高效来源 内置自动缩放管道。
Metrics Server 从 Kubelet 收集资源指标,并通过 Metrics API 在 Kubernetes apiserver 中公开它们,供 Horizontal Pod Autoscaler 和 Vertical Pod Autoscaler 使用。
Metrics Server offers:
-
A single deployment that works on most clusters (see Requirements)
部署一个deploy可以满足绝大多数的集群需求
-
Fast autoscaling, collecting metrics every 15 seconds.
每隔15s收集一次指标信息
-
Resource efficiency, using 1 mili core of CPU and 2 MB of memory for each node in a cluster.
资源消耗。集群中每个node 需要1毫核 cpu和 2MB 内存
-
Scalable support up to 5,000 node clusters.
支持单个集群5000node
安装要求
-
kube-apiserver 开启聚合
aggregation layer: 允许apiserver 扩展额外的apis
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --requestheader-allowed-names=front-proxy-client - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --requestheader-extra-headers-prefix=X-Remote-Extra- - --requestheader-group-headers=X-Remote-Group - --requestheader-username-headers=X-Remote-User
-
开启webhook认证
-
Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing to Metrics Server)
--kubelet-insecure-tls
-
Container runtime must implement a container metrics RPCs (or have cAdvisor support)
-
Network should support following communication:
- Control plane to Metrics Server. Control plane node needs to reach Metrics Server's pod IP and port 10250 (or node IP and custom port if is enabled). Read more about control plane to node communication.
hostNetwork
- Metrics Server to Kubelet on all nodes. Metrics server needs to reach node address and Kubelet port. Addresses and ports are configured in Kubelet and published as part of Node object. Addresses in and port in field (default 10250). Metrics Server will pick first node address based on the list provided by command line flag (default in manifests).
.status.addresses``.status.daemonEndpoints.kubeletEndpoint.port``kubelet-preferred-address-types``InternalIP,ExternalIP,Hostname
- Control plane to Metrics Server. Control plane node needs to reach Metrics Server's pod IP and port 10250 (or node IP and custom port if is enabled). Read more about control plane to node communication.
版本选择
指标服务器 | 指标 API 组/版本 | 支持的 Kubernetes 版本 |
---|---|---|
0.6.x | metrics.k8s.io/v1beta1 |
1.19+ |
0.5.x | metrics.k8s.io/v1beta1 |
*1.8+ |
0.4.x | metrics.k8s.io/v1beta1 |
*1.8+ |
0.3.x | metrics.k8s.io/v1beta1 |
1.8-1.21 |
*低于 v1.16 的 Kubernetes 版本需要传递命令行标志--authorization-always-allow-paths=/livez,/readyz
[root@allinone dashboard]# k top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
安装
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.2/high-availability-1.21+.yaml
在args中增加--kubelet-insecure-tls=true
- args:
- --kubelet-insecure-tls=true
验证
kubectl top pod -A | sort -k4 -rn
参考:
用 kubeadm 进行管理 | Kubernetes
Cgroup(Control Group)主要用于限制和隔离一组进程对系统资源的使用(包括CPU、内存、block I/O、网络带宽等)。
cgroupfs就是Cgroup的一个接口的封装,cgroupfs挂载在ls /sys/fs/cgroup/
Systemd也是对于Cgroup接口的一个封装
cgroupfs
https://blog.csdn.net/cymm_liu/article/details/106677997
bridge-nf-call-iptables
https://cloud.tencent.com/developer/article/1828060
https://blog.csdn.net/tycoon1988/article/details/40826235