containerd 容器基础环境组件的搭建
1 基础环境说明
(1)本次所有部署软件版本说明
| 软件名称 | 版本号 |
| 操作系统内核(后续升级为 lt-5.4.278) |
CentOS 7.9.2009 (3.10.0-1160.el7) 1c1GB 20GB |
| containerd | v1.6.6 |
| cfssl | v1.6.1 |
| cni | v1.1.1 |
| crictl | v1.24.2 |
| nerdctl | 1.7.6 |
| buildkit | v0.14.1 |
(2)系统参数修改
<1>/etc/security/limits.conf
ulimit -SHn 65535 cat >> /etc/security/limits.conf <<EOF * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * seft memlock unlimited * hard memlock unlimitedd EOF
(3) libseccomp软件升级
<1>libseccomp版本检查分析
1>查看现有libseccomp版本
rpm -qa | grep libseccomp
<2>问题解决
1>卸载低版本libseccomp
rpm -e libseccomp-2.3.1-4.el7.x86_64 --nodeps
rpm -qa | grep libseccomp
2>安装高版本libseccomp
wget https://vault.centos.org/centos/8/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm
rpm -ivh libseccomp-2.5.1-1.el8.x86_64.rpm
rpm -qa | grep libseccomp
2 安装Containerd作为Runtime
(1)环境初始化
<1>安装一些必备工具
yum -y install wget jq psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git network-scripts tar curl lrzsz chrony sshpass
<2>网络配置
cat > /etc/NetworkManager/conf.d/calico.conf << EOF [keyfile] unmanaged-devices=interface-name:cali*;interface-name:tunl* EOF systemctl restart NetworkManager
<3>内核部分
1>添加启用源
yum -y install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm sed -i "s@mirrorlist@#mirrorlist@g" /etc/yum.repos.d/elrepo.repo sed -i "s@elrepo.org/linux@mirrors.tuna.tsinghua.edu.cn/elrepo@g" /etc/yum.repos.d/elrepo.repo wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo yum clean all && yum makecache #查看可用安装包 yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
2>升级内核至 4.18 版本以上(内核的升级建议一台一台进行升级) 说明: 稳定版 kernel-ml，如需更新长期维护版本 kernel-lt(此处选择)
yum -y --enablerepo=elrepo-kernel install kernel-lt
3>查看已安装那些内核
rpm -qa | grep kernel
4>查看默认内核: grubby --default-kernel
5>若不是最新的使用命令设置
grubby --set-default /boot/vmlinuz-5.4.278-1.el7.elrepo.x86_64
6>重启生效: reboot
<4>安装 ipvsadm(此处安装是为了后续给k8s使用)
说明：所有节点配置ipvs模块，在内核 4.19+版本 nf_conntrack_ipv4 已经改为nf_conntrack，4.18 以下使用 nf_conntrack_ipv4 即可
yum -y install ipvsadm ipset sysstat conntrack libseccomp cat >> /etc/modules-load.d/ipvs.conf <<EOF ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip EOF systemctl restart systemd-modules-load.service
检查: lsmod | grep -e ip_vs -e nf_conntrack
ip_vs_sh 16384 0 ip_vs_wrr 16384 0 ip_vs_rr 16384 0 ip_vs 155648 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 147456 1 ip_vs nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs nf_defrag_ipv4 16384 1 nf_conntrack libcrc32c 16384 3 nf_conntrack,xfs,ip_vs
<5>修改内核参数
cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 65536 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 net.ipv6.conf.all.disable_ipv6 = 0 net.ipv6.conf.default.disable_ipv6 = 0 net.ipv6.conf.lo.disable_ipv6 = 0 net.ipv6.conf.all.forwarding = 1 EOF sysctl --system
(2)安装说明
<1>下载
wget https://github.com/containerd/containerd/releases/download/v1.6.6/cri-containerd-cni-1.6.6-linux-amd64.tar.gz
<2>创建 cni 插件所需目录
mkdir -p /etc/cni/net.d mkdir -p /opt/cni/bin
<3>解压cni二进制包
tar -xf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin/ tar -xzf cri-containerd-cni-1.6.6-linux-amd64.tar.gz -C /
<4>创建服务启动文件
cat > /etc/systemd/system/containerd.service <<EOF [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=infinity TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target EOF
(3)配置 Containerd 所需的模块
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf overlay br_netfilter EOF #加载模块 systemctl restart systemd-modules-load.service
(4)创建 Containerd 的配置文件
<1>配置修改
mkdir -p /etc/containerd containerd config default | tee /etc/containerd/config.toml #修改 Containerd 的配置文件 sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml cat /etc/containerd/config.toml | grep SystemdCgroup
(5)启动并设置为开机启动
systemctl daemon-reload && systemctl enable --now containerd
(6)配置 crictl 客户端连接的运行时位置
<1>下载
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.24.2/crictl-v1.24.2-linux-amd64.tar.gz tar -xf crictl-v1.24.2-linux-amd64.tar.gz -C /usr/bin/
<2>生成配置文件
cat > /etc/crictl.yaml <<EOF runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: false EOF #测试 systemctl restart containerd crictl info
(7)开机检查启动项
systemctl status NetworkManager && systemctl status systemd-modules-load.service systemctl daemon-reload && systemctl status containerd crictl info
3 nerdctl和buildctl安装
(1)安装nerdctl
<1>安装
wget https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-1.7.6-linux-amd64.tar.gz tar -xf nerdctl-1.7.6-linux-amd64.tar.gz -C /usr/local/bin/
<2>验证 nerdctl -v
(2)安装buildctl
<1>安装
wget https://github.com/moby/buildkit/releases/download/v0.14.1/buildkit-v0.14.1.linux-amd64.tar.gz tar -xf buildkit-v0.14.1.linux-amd64.tar.gz mv bin/* /usr/local/bin/
buildctl -v
<2>提供buildkit.socket文件
cat > /etc/systemd/system/buildkit.socket <<EOF [Unit] Description=BuildKit Documentation=https://github.com/moby/buildkit [Socket] ListenStream=%t/buildkit/buildkitd.sock SocketMode=0660 [Install] WantedBy=sockets.target EOF
<3>配置启动脚本
cat > /etc/systemd/system/buildkit.service <<EOF [Unit] Description=BuildKit Requires=buildkit.socket After=buildkit.socket Documentation=https://github.com/moby/buildkit [Service] ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true --config=/etc/buildkit/buildkitd.toml [Install] WantedBy=multi-user.target EOF
<4>配置配置文件(此处是要使用代理才使用的，如果不需要则不配置上述启动文件的--config即可)
mkdir /etc/buildkit mkdir /etc/nerdctl mkdir /etc/cert cat > /etc/buildkit/buildkitd.toml <<EOF debug = true insecure-entitlements = ["network.host", "security.insecure"] [registry."www.xxxxx.com"] ca=["/etc/cert/www.xxxxx.com.crt"] insecure = true EOF cat > /etc/nerdctl/nerdctl.toml <<EOF namespace = "k8s.io" debug = false debug_full = false insecure_registry = true EOF
<5>启动
systemctl daemon-reload && systemctl enable buildkit --now
systemctl status buildkit
<6>检查
ll /run/buildkit/buildkitd.sock
