k8s离线部署v1.28.0版本（基于docker容器）

1.环境配置

主机名	配置	磁盘大小	操作系统	ip地址
k8s-master	2c4g	50g	centos7.6	192.168.100.194
k8s-node1	2c4g	50g	centos7.6	192.168.100.195
k8s-node2	2c4g	50g	centos7.6	192.168.100.196
yum	2c4g	50g	centos7.6	192.168.100.201

2.必要环境准备

1）关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

2）关闭selinux

setenforce 0 临时

sed -i 's/enforcing/disabled/' /etc/selinux/config 永久

3）关闭swap

swapoff -a临时

sed -ri 's/.*swap.*/#&/' /etc/fstab 永久

4）设置主机名

hostnamectl set-hostname k8s-master

hostnamectl set-hostname k8s-node1

hostnamectl set-hostname k8s-node2
5）master添加hosts

cat >> /etc/hosts << EOF

192.168.100.194 k8s-master

192.168.100.195 k8s-node1

192.168.100.196 k8s-node2

EOF

6）调整内核参数，三台服务器将桥接的ipv4流量传递到iptables链

cat > /etc/sysctl.d/k8s.conf << EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

sysctl --system #生效

3.安装docker（3台都执行）--联网机器下载

--下载docker环境yum源
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
--只下载不安装配置，创建本地yum仓库
yum install docker-ce --downloadonly --downloaddir=/var/www/html/docker 
createrepo  /var/www/html/docker
--3台yum客户端配置
[docker]
name=docker
baseurl=http://192.168.100.201/docker
gpgcheck=0
enabled=1
--3台yum客户端安装docker，不指定版本就是最新版
yum -y install docker-ce 
--配置docker加速
cat > /etc/docker/daemon.json << EOF
{
    "registry-mirrors": [
        "https://docker.m.daocloud.io"
    ]
}
EOF
systemctl enable docker && systemctl start docker 设置开机自启并启动docker

4.安装cri-docker(docker与k8s通信的中程序：翻译官)

说明：从1.24版本开始k8s默认容器已经不是docker，如果要通过docker作为k8s的容器运行时需要安装组件，进行通信

?

1 2 3 4 5 6 7 8 9 10

# 下载 wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.2/cri-dockerd-0.3.2-3.el7.x86_64.rpm # 安装 rpm -ivh cri-dockerd-0.3.2-3.el7.x86_64.rpm # 修改cri-docker镜像地址 vi /usr/lib/systemd/system/cri-docker.service ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9   # 启动 systemctl enable cri-docker && systemctl start cri-docker

5.安装配置k8s的yum源

---在yum服务器（访问外网）下载不安装对应的包
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
---下载相关包到本地
yum install kubelet-1.28.0 kubeadm-1.28.0 kubectl-1.28.0 --downloadonly --downloaddir=/var/www/html/k8s
---创建本地yum仓库
createrepo /var/www/html/k8s
---在3台服务器上配置yum客户端
[k8s]
name=k8s
baseurl=http://192.168.100.201/k8s
gpgcheck=0
enabled=1

 

6.安装kubctl、kubeadm、kubelet（3台都执行）

?

1 2 3

kubeadm：集群的初始化部署和升级 kubectl：命令行工具，用于向APIServer发送指令，创建pod等资源。 kubelet ：是在每个节点上运行的代理，它负责接受主节点上的 API Server下放的指令和监控pod

yum install kubelet-1.28.0 kubeadm-1.28.0 kubectl-1.28.0 -y 

systemctl enable kubelet 设置开机自启动

7.k8s镜像下载--联网机器下载

--获取需要下载镜像的列表

kubeadm config images list 

 

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

--镜像下载 docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.0 docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.0 docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.0 docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.28.0 docker pull registry.aliyuncs.com/google_containers/etcd:3.5.9-0 docker pull registry.aliyuncs.com/google_containers/coredns:v1.10.1 docker pull registry.aliyuncs.com/google_containers/pause:3.9 --镜像压缩 docker save -o kube-apiserver.tar registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.0 docker save -o kube-controller-manager.tar registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.0 docker save -o kube-scheduler.tar registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.0 docker save -o kube-proxy.tar registry.aliyuncs.com/google_containers/kube-proxy:v1.28.0 docker save -o pause.tar registry.aliyuncs.com/google_containers/pause:3.9 docker save -o etcd.tar registry.aliyuncs.com/google_containers/etcd:3.5.9-0 docker save -o coredns.tar registry.aliyuncs.com/google_containers/coredns:v1.10.1 --镜像加载 docker load -i kube-apiserver.tar docker load -i kube-controller-manager.tar docker load -i kube-scheduler.tar docker load -i kube-proxy.tar docker load -i pause.tar docker load -i etcd.tar docker load -i coredns.tar

 8.部署k8s（master节点操作）

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

# apiserver-advertise-address  配置k8s apiserver地址，用于监听、响应其他节点请求 # --service-cidr=10.96.0.0/12 配置k8s Service的IP范围 # --pod-network-cidr=10.244.0.0/16 配置k8s pod的IP范围 kubeadm init \   --apiserver-advertise-address=192.168.100.194 \   --image-repository registry.aliyuncs.com/google_containers \   --kubernetes-version v1.28.0 \   --service-cidr=10.96.0.0/12 \   --pod-network-cidr=10.244.0.0/16 \   --cri-socket=unix:///var/run/cri-dockerd.sock \   ##--ignore-preflight-errors=all   #忽略错误，不然一直拉取外网镜像   --安装成功后master节点执行 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

9.node节点加入到集群

?

1 2 3

kubeadm join 10.44.100.194:6443 --token 6xkje6.g53th6yjstzv79e2 --discovery-token-ca-cert-hash sha256:803c78010edaa35ab481e05a1493ed832294cbfb45982fe2f82314a499d2fe5a  --cri-socket unix:///var/run/cri-dockerd.sock  token有效期24小时，过了后，重新生成token： kubeadm token create --print-join-command

此时，查看集群节点，还没ready，需要安装网络插件进行通信

 10.安装flannel网络插件，让node间通信

cat kube-flannel.yml | grep image 下载所需镜像

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
docker pull docker.m.daocloud.io/flannel/flannel:v0.25.6
docker pull docker.m.daocloud.io/flannel/flannel-cni-plugin:v1.5.1-flannel2
保存镜像到tar包
docker save -o flannel_v0.25.6.tar docker.m.daocloud.io/flannel/flannel:v0.25.6
docker save -o flannel-cni-plugin_v1.5.1-flannel2.tar docker.m.daocloud.io/flannel/flannel-cni-plugin:v1.5.1-flannel2
导入镜像-所有节点
docker load -i flannel_v0.25.6.tar
docker load -i flannel-cni-plugin_v1.5.1-flannel2.tar
安装flannel
kubectl apply -f kube-flannel.yml

 

查看最终的成功状态

 

附安装中的错误处理：

1）init初始化报错

[ERROR CRI]: container runtime is not running: output: time="2024-09-24T09:56:19+08:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint "unix:///var/run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

原因：k8s自从v1.24后,默认容器运行时改成了containerd,containerd是一个CRI（containner runtime interface）组件，在容器运行时调用containerd组件来创建、运行、销毁容器

解决：disabled_plugins = ["cri"] 改为 disabled_plugins = []  &&然后重启systemctl restart containerd 

2）k8s安装失败后的重置

kubeadm reset

rm -rf /etc/kubernetes/*

rm -rf /root/.kube

3）Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket'

原因： 表明在环境中有多个容器运行时接口（containner runtime interface），k8s不确定用哪一个

解决：在kubeadm init时指定要使用的cri端点 --cri-socket unix:///var/run/cri-dockerd.sock

4）部署flannel 提示Init:ImagePullBackOff

原因：拉取失败，连接不到docker.io

编辑kube-flannel.yml文件，把image字段对应的docker.io改为docker.m.daocloud.io

重新部署：kubectl delete -f kube-flannel.yml&&kubectl apply -f kube-flannel.yml

5）从节点执行kubectl命令提示

E0927 09:56:12.002974 22410 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused

原因：k8s默认从~/.kube/config 配置文件获取访问kube-apiserver地址、证书、用户名等信息，如果没有配置文件会默认读取默认值localhost:8080,而本机的localhost:8080没有服务报错。

解决办法：mkdir ~/.kube

cp /etc/kubernetes/kubelet.conf ~/.kube/config