Kubernetes1.24还想使用Docker怎么办？

1：环境

Kubernetes-master-1	10.0.0.10
Kubernetes-worker-1	10.0.0.11
Kubernetes-worker-2	10.0.0.12

2：Kubernetes1.24版本改动

2022 年 5 月 3 日，Kubernetes 1.24 正式发布，在新版本中，我们看到 Kubernetes 作为容器编排的事实标准，正愈发变得成熟，有 12 项功能都更新到了稳定版本，同时引入了很多实用的功能，例如 StatefulSets 支持批量滚动更新，NetworkPolicy 新增 NetworkPolicyStatus 字段方便进行故障排查等

Kubernetes 1.24 改动
Kubernetes 正式移除对 Dockershim 的支持，讨论很久的 “弃用 Dockershim” 也终于在这个版本画上了句号。
想要清楚的了解docker 和 k8s 的关系，可以参考下这篇文章： https://i4t.com/5435.html

2.1：Kubernetes1.24 之前

2.2：Kubernetes1.24 之后

如还想继续在k8s中使用docker，需要自行安装cri-dockerd 组件； 不然就使用containerd 

3：开始部署

3.1：基础配置

# 主机名配置 （三台主机操作）
hostnamectl set-hostname kubernetes-master-1          # master节点
hostnamectl set-hostname kubernetes-worker-1          # worker-1节点
hostnamectl set-hostname kubernetes-worker-2          # worker-2节点

# host解析 （三台节点操作）
cat << EOF>> /etc/hosts
10.0.0.10   kubernetes-master-1
10.0.0.11   kubernetes-worker-1
10.0.0.12   kubernetes-worker-2
EOF

# 关闭SWAP分区 （三台节点操作）
# 临时关闭
swapoff -a
#永远关闭swap分区，需要重启操作系统
sed -i '/ swap / s/^(.*)$/#1/g' /etc/fstab

# 防火墙配置 （三台节点操作）
systemctl disable firewalld --now

# SELINUX配置 （三台节点操作）
# 临时关闭
setenforce 0
#永久生效
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

# 时间同步配置 （三台节点操作）
# crontab -l
0 */1 * * * /usr/sbin/ntpdate ntp.aliyun.com
# 设置上海时区，东八区
timedatectl set-timezone Asia/Shanghai

# 升级操作系统内核 （三台节点操作）
# 导入elrepo gpg key
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# 安装elrepo YUM源仓库
yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
# 安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本
yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64
# 设置grub2默认引导为0
grub2-set-default 0
# 重新生成grub2引导文件
grub2-mkconfig -o /boot/grub2/grub.cfg
# 更新后，需要重启，使用升级的内核生效
reboot
# 查看内核
uname -r
# 配置内核转发及网桥过滤
cat << EOF>>/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF
# 使它生效
sysctl -p /etc/sysctl.d/k8s.conf
# 加载br_netfilter模块
modprobe br_netfilter
# 查看是否加载
lsmod | grep br_netfilter
# 安装ipset及ipvsadm
yum -y install ipset ipvsadm
# 配置ipvsadm模块加载方式.添加需要加载的模块
cat << EOF>> /etc/sysconfig/modules/ipvs.module
modprobe -- ip_vs
modprobe -- ip_vs_sh
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- nf_conntrack
EOF
# 赋予执行权限并执行。检查是否加载
chmod 755 /etc/sysconfig/modules/ipvs.module &&  /etc/sysconfig/modules/ipvs.module
lsmod | grep ip

3.2：Docker部署

因为Docker要使用到二进制，所以，我们完全可以将/var/lib/docker配置在一个单独的盘上，我这里给每台机器都加上了一块 50G磁盘
fdisk -l
---
Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


# 格始化磁盘
mkfs.xfs /dev/sdb
# 创建docker工作目录
mkdir /var/lib/docker
# 写入挂载信息到fstab中，永久挂载
echo "/dev/sdb /var/lib/docker xfs defaults 0 0" >>  /etc/fstab
# 使fstab挂载生效
mount -a
# 查看磁盘挂载
df -Th | grep sdb
# 安装一些必要工具
yum install -y yum-utils device-mapper-persistent-data lvm2
# Docker安装(下载源)
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
# 安装源里最新版本
yum install docker-ce -y
# 启动Docker服务
systemctl enable docker --now
# 配置docker加速，修改cgroup方式
cat << EOF>> /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://6ze43vnb.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# 重启Docker
systemctl daemon-reload && systemctl restart docker

# cri-dockerd安装
# 下载cri-dockerd 二进制文件
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.1/cri-dockerd-0.2.1.amd64.tgz
# 解压并移动
tar xf cri-dockerd-0.2.1.amd64.tgz
cp cri-dockerd/cri-dockerd /usr/bin/
chmod +x /usr/bin/cri-dockerd

# 配置启动文件
cat << EOF>> /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint=unix:///var/run/cri-docker.sock --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --image-pull-progress-deadline=30s --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7 --docker-endpoint=unix:///var/run/docker.sock --cri-dockerd-root-directory=/var/lib/docker
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

EOF

# 生成socket 文件
cat << EOF>> /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=/var/run/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

EOF

# 也可以直接下载https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd 注意，需要修改cri-docker.service 中 ExecStart 启动参数，另外cri-docker.socket 文件中也要指明 ListenStream 的地址cri-dockerd.sock

# 启动cri-docker
systemctl daemon-reload
systemctl start cri-docker
systemctl enable cri-docker
systemctl status cri-docker

3.3：Kubernetes 1.24.1部署

# kubernetes加速源准备
cat << EOF>> /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 集群软件安装
# 查看1.24的可用版本
yum list  kubeadm  kubelet kubectl --showduplicates | sort -r | grep 1.24
# 安装最新版
yum install  kubeadm  kubelet kubectl -y
# 安装后查看版本
kubeadm version
# 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动
systemctl enable kubelet

# 配置kubelet 为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。
cat <<EOF > /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

# 初始化配置文件 （Master 节点操作）
kubeadm config print init-defaults > kubeadm.yaml

# 修改配置文件
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.0.10
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/cri-docker.sock
  imagePullPolicy: IfNotPresent
  name: kubernetes-master-1
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.24.1
networking:
  dnsDomain: cluster.local
  serviceSubnet: 100.1.0.0/12
  podSubnet: 200.1.0.0/16
scheduler: {}

---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

# 集群镜像准备
# 查看需要使用的镜像列表,若无问题，将得到如下列表
kubeadm config images list --config kubeadm.yaml
# 提前下载镜像到本地
kubeadm config images pull --config kubeadm.yaml
# 集群初始化
kubeadm init --config=kubeadm.yaml
# 重置 如果有需要，必须要指定--cri-socket，不然会报错
kubeadm reset --cri-socket unix:///var/run/cri-docker.sock

# 安装完成
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  echo "source <(kubectl completion bash)" >> ~/.bashrc
  source ~/.bashrc
Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.10:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:46c0a14f577986ebeeca1a75d79479d6db63afc63c5f8cbd1ecadcef0fa1b856 
	
# 加入集群时记得指定一下 cri
kubeadm join 10.0.0.10:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:46c0a14f577986ebeeca1a75d79479d6db63afc63c5f8cbd1ecadcef0fa1b856 --cri-socket unix:///var/run/cri-docker.sock

# 查看集群
[root@kubernetes-master-1 ~]# kubectl get nodes -owide
NAME                  STATUS     ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
kubernetes-master-1   NotReady   control-plane   4m48s   v1.24.1   10.0.0.10     <none>        CentOS Linux 7 (Core)   5.18.2-1.el7.elrepo.x86_64   docker://20.10.17
kubernetes-worker-1   NotReady   <none>          12s     v1.24.1   10.0.0.11     <none>        CentOS Linux 7 (Core)   5.18.2-1.el7.elrepo.x86_64   docker://20.10.17
kubernetes-worker-2   NotReady   <none>          48s     v1.24.1   10.0.0.12     <none>        CentOS Linux 7 (Core)   5.18.2-1.el7.elrepo.x86_64   docker://20.10.17

# 部署calico网络
[root@kubernetes-master-1 ~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

[root@kubernetes-master-1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}   
scheduler            Healthy   ok