K8S minikube
Minikube是Kubernetes官方推荐学习使用的快速搭建Kubernetes集群的工具，它允许将Master和Node的组件运行在同一台物理主机上。
Minikube工具实际上也是通过Kubeadm快速构建的Kubernetes集群。
Minikube搭建Kubernetes集群只需要一台物理主机。
快速部署集群v1.24.0+（单机版）
参考文档：https://minikube.sigs.k8s.io/docs/start/
系统版本：CentOS Linux release 7.6.1810 (Core)
软件版本：Docker-ce-3:23.0.1-1.el7、Kubernetes-v1.24.0
硬件要求：建议最低2核4GB
1、配置系统环境
# 配置主机名
[root@localhost ~]# echo 'minikube' >/etc/hostname
[root@localhost ~]# cat /etc/hostname |xargs hostname
[root@localhost ~]# bash
# 配置主机解析
[root@localhost ~]# cat <<EOF >> /etc/hosts
172.16.254.136 minikube
EOF
# 关闭防火墙
[root@minikube ~]# systemctl stop firewalld
[root@minikube ~]# systemctl disable firewalld
# 关闭SELinux
[root@minikube ~]# setenforce 0
[root@minikube ~]# sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux
# 关闭SWAP交换分区
[root@minikube ~]# swapoff -a
# 启用br_netfilter二层流量过滤功能
[root@minikube ~]# modprobe br_netfilter
[root@minikube ~]# lsmod | grep br_netfilter
br_netfilter 22256 0
bridge 151336 1 br_netfilter
[root@minikube ~]# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
[root@minikube ~]# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@minikube ~]# sysctl --system
2、安装Docker
# 配置YUM-Docker源
# Docker-YUM源由阿里巴巴开源镜像网提供。
[root@minikube ~]# yum -y install epel-release.noarch yum-utils
[root@minikube ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装依赖
[root@minikube ~]# yum -y install device-mapper-persistent-data lvm2
# 查看能安装的Docker版本
[root@minikube ~]# yum list docker-ce.x86_64 --showduplicates | sort -r
# 安装Docker
[root@minikube ~]# yum -y install docker-ce-3:23.0.1-1.el7
# 启动Docker服务
[root@minikube ~]# systemctl start docker
[root@minikube ~]# systemctl enable docker
[root@minikube ~]# systemctl status docker
# 配置Docker使用国内镜像源
[root@minikube ~]# cat <<EOF > /etc/docker/daemon.json
{
"registry-mirrors": ["http://hub-mirror.c.163.com"]
}
EOF
# 重启Docker服务
[root@minikube ~]# systemctl restart docker
[root@minikube ~]# systemctl status docker
3、安装kubectl、kubelet、kubernets-cni
[root@minikube ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@minikube ~]# yum install kubectl-1.24.0 kubelet-1.24.0-0 kubernetes-cni-1.2.0-0 -y --nogpgcheck
4、安装CRI-Docker、Crictl
安装最新版本的Kubernetesv1.24.0+，需要额外安装CRI-Docker
CRI-Docker为Kubernetes提供一个操作Docker的运行时接口。
Crictl用于Kubelet容器运行时接口 (CRI) 的CLI和验证工具。
参考文档：
https://github.com/Mirantis/cri-dockerd
https://github.com/kubernetes-sigs/cri-tools
# 安装CRI-Docker
[root@minikube ~]# wget https://ghproxy.com/https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd-0.3.1.amd64.tgz
[root@minikube ~]# tar xzvf cri-dockerd-0.3.1.amd64.tgz
cri-dockerd/
cri-dockerd/cri-dockerd
[root@minikube ~]# cp cri-dockerd/cri-dockerd /usr/bin/
# 配置CRI-Docker作为SYSTEM系统服务
# 参考文件: https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd
# 创建cri-docker.service文件
# 这边启动参数需要设置为ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
[root@minikube ~]# vim /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
# 创建cri-docker.socket文件
[root@minikube ~]# vim /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
# 启动CRI-Docker服务并设置为开机自启
[root@minikube ~]# systemctl daemon-reload
[root@minikube ~]# systemctl restart cri-docker
[root@minikube ~]# systemctl status cri-docker
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/cri-docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2023-01-31 21:32:54 EST; 4s ago
Docs: https://docs.mirantis.com
Main PID: 13701 (cri-dockerd)
Tasks: 9
Memory: 14.2M
CGroup: /system.slice/cri-docker.service
└─13701 /usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Start docker client with request timeout 0s"
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Hairpin mode is set to none"
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Loaded network plugin cni"
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Docker cri networking managed by network plugin cni"
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Docker Info: &{ID:I32G:GCJA:CKTO:5ZIC:2AED:6KYI...] [Nativ
Jan 31 21:32:54 minikube systemd[1]: Started CRI Interface for Docker Application Container Engine.
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Setting cgroupDriver cgroupfs"
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Docker cri received runtime config &RuntimeConf...dr:,},}"
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Starting the GRPC backend for the Docker CRI interface."
Jan 31 21:32:54 minikube cri-dockerd[13701]: time="2023-01-31T21:32:54-05:00" level=info msg="Start cri-dockerd grpc backend"
Hint: Some lines were ellipsized, use -l to show in full.
[root@minikube ~]# systemctl enable cri-docker
Created symlink from /etc/systemd/system/multi-user.target.wants/cri-docker.service to /usr/lib/systemd/system/cri-docker.service.
# 安装crictl
[root@minikube ~]# VERSION="v1.26.0"
[root@minikube ~]# wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
[root@minikube ~]# sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
[root@minikube ~]# rm -f crictl-$VERSION-linux-amd64.tar.gz
[root@minikube ~]# ln /usr/local/bin/crictl /usr/bin
# 或通过YUM安装crictl[推荐]
[root@minikube ~]# yum -y install epel-release.noarch
[root@minikube ~]# yum -y install cri-tools
5、安装Minikube
[root@minikube ~]# curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
[root@minikube ~]# install minikube-linux-amd64 /usr/local/bin/minikube
[root@minikube ~]# minikube version
6、使用Minikube创建一个Kubernetes单点集群
[root@minikube ~]# minikube start --vm-driver=none --image-mirror-country='cn' --image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' --kubernetes-version=v1.24.0 --cri-socket='/var/run/cri-dockerd.sock'
# --vm-driver=none 表示使用Linux本机作为运行环境。
# --kubernetes-version=xxx 表示指定Kubernetes的版本。
# --image-mirror-country='cn' 表示使用中国地区的镜像。
7、查看Kubernetes集群运行情况
[root@minikube ~]# minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
timeToStop: Nonexistent
8、开启Dashboard
我们可以使用一个WEB UI图形化界面查看和管理Kubernetes集群。
# 启用仪表盘
# 使用Minikube启动仪表盘后，会打印一个URL，我们可以通过URL访问到仪表盘。
[root@minikube ~]# nohup minikube dashboard &
Opening http://127.0.0.1:37008/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...
# 外部访问到仪表盘
# 默认情况下开放的地址只允许本地访问，若想要外部访问到仪表盘，则需要暴露一个代理，允许外部访问。
# 由于URL路径较长，要使用Google浏览器访问，或者可以使用Nginx反向代理缩短URL长度。
[root@minikube ~]# nohup kubectl proxy --port="8001" --address="172.16.254.136" --accept-hosts="^.*" &
http://172.16.254.136:8001/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/
使用Minikube管理集群
集群管理
运行集群
[root@minikube ~]# minikube start --vm-driver=none --kubernetes-version=v1.24.0 --image-mirror-country='cn'
# --vm-driver=none 表示使用Linux本机作为运行环境。
# --kubernetes-version=xxx 表示指定Kubernetes的版本。
# --image-mirror-country='cn' 表示使用中国地区的镜像。
删除集群
[root@minikube ~]# minikube stop
[root@minikube ~]# minikube delete
[root@minikube ~]# rm -rf /root/.minikube
停止集群
[root@minikube ~]# minikube stop
重启集群
[root@minikube ~]# minikube stop && minikube start
查看集群各组件运行情况
[root@minikube ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6967fb4995-4b4c5 1/1 Running 0 2m
kube-system coredns-6967fb4995-sbg2z 1/1 Running 0 2m
kube-system etcd-minikube 1/1 Running 0 55s
kube-system kube-addon-manager-minikube 1/1 Running 0 44s
kube-system kube-apiserver-minikube 1/1 Running 0 52s
kube-system kube-controller-manager-minikube 1/1 Running 0 55s
kube-system kube-proxy-md5dg 1/1 Running 0 2m
kube-system kube-scheduler-minikube 1/1 Running 0 70s
kube-system kubernetes-dashboard-95564f4f-fkhv5 1/1 Running 0 119s
kube-system storage-provisioner 1/1 Running 0 118s
插件管理
Minikube有一组内置的加载项，可以在本地Kubernetes环境中启用、禁用和打开。
这些加载项属于一些第三方Kubernetes附件（插件），可以实现一些高级功能。
查看可开启的插件列表
[root@minikube ~]# minikube addons list
- addon-manager: enabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- gvisor: disabled
- heapster: disabled
- ingress: disabled
- logviewer: disabled
- metrics-server: disabled
- nvidia-driver-installer: disabled
- nvidia-gpu-device-plugin: disabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled
- storage-provisioner-gluster: disabled
启用插件
heapster是Kubernetes常用的一个附件（插件），启用它我们可以查看Pod对象资源使用情况，它是一个任务管理器。
ingress是Kubernetes中必备附件，用于公开应用，实现基于HTTP/S七层的负载均衡。
[root@minikube ~]# minikube addons enable heapster
* heapster was successfully enabled
[root@minikube ~]# kubectl top pod
NAME CPU(cores) MEMORY(bytes)
hello-node-56ddd6c85d-wbspf 0m 9Mi
[root@minikube ~]# minikube addons enable ingress
查看插件是否启动成功
启用附件，会在名称空间（kube-system）下创建一个吊舱资源对象（Pod）和一个服务资源对象（Service），并在Pod中运行对应的容器。
[root@minikube ~]# kubectl get pod,svc -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-6967fb4995-4b4c5 1/1 Running 0 5h41m
pod/coredns-6967fb4995-sbg2z 1/1 Running 0 5h41m
pod/etcd-minikube 1/1 Running 0 5h40m
pod/heapster-d8bsq 1/1 Running 0 3m31s
pod/influxdb-grafana-dcpqj 2/2 Running 0 3m31s
pod/kube-addon-manager-minikube 1/1 Running 0 5h40m
pod/kube-apiserver-minikube 1/1 Running 0 5h40m
pod/kube-controller-manager-minikube 1/1 Running 0 5h40m
pod/kube-proxy-md5dg 1/1 Running 0 5h41m
pod/kube-scheduler-minikube 1/1 Running 0 5h40m
pod/kubernetes-dashboard-95564f4f-fkhv5 1/1 Running 0 5h41m
pod/storage-provisioner 1/1 Running 0 5h41m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/heapster ClusterIP 10.103.127.8 <none> 80/TCP 3m31s
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 5h41m
service/kubernetes-dashboard ClusterIP 10.97.77.8 <none> 80/TCP 5h41m
service/monitoring-grafana NodePort 10.104.40.70 <none> 80:30002/TCP 3m31s
service/monitoring-influxdb ClusterIP 10.103.102.119 <none> 8083/TCP,8086/TCP 3m31s
禁用插件
[root@minikube ~]# minikube addons disable heapster
附录：重启集群中的发生问题
[kubelet-check] Initial timeout
报错信息：
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock logs CONTAINERID'
stderr:
W0215 03:40:32.375153 99610 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
查看kubelet报错信息：
[root@minikube ~]# journalctl -xeu kubelet |more
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.149848 64597 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandb
ox\" for \"etcd-minikube_kube-system(a75b77b0a9e517a0cac04559a1c583ec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-minikub
e_kube-system(a75b77b0a9e517a0cac04559a1c583ec)\\\": rpc error: code = Unknown desc = failed pulling image \\\"registry.k8s.io/pause:3.6\\\": Error resp
onse from daemon: Head \\\"https://asia-northeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6\\\": dial tcp 142.250.157.82:443: con
nect: connection refused\"" pod="kube-system/etcd-minikube" podUID=a75b77b0a9e517a0cac04559a1c583ec
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.230355 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.330444 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.430833 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.531775 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.632183 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.733334 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.834118 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:32 minikube kubelet[64597]: E0215 05:14:32.935064 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:33 minikube kubelet[64597]: E0215 05:14:33.035527 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
Feb 15 05:14:33 minikube kubelet[64597]: E0215 05:14:33.135691 64597 kubelet.go:2419] "Error getting node" err="node \"minikube\" not found"
原因分析：
是由于v1.24.0后启用了CRI sandbox(pause) image的配置支持。通过kubeadm init –image-repository设置的镜像地址，不再会传递给cri运行时去下载pause镜像。而是需要在cri运行时的配置文件中设置。
问题处理：
使用VPN网络FQ后拉取镜像会成功。或配置CRI运行时设置使用的国内镜像即可：
[root@minikube ~]# vim /etc/containerd/config.toml
# 追加以下内容
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.aliyuncs.com/k8sxio/pause:3.6"
[root@minikube ~]# systemctl restart kubelet
附录：在Minikube中使用Flannel网络
在构建之前，请先删除原有集群！
1.安装Kubernets-cni
[root@minikube ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@minikube ~]# yum install kubernetes-cni -y
2.启动一个使用Flannel网络的Kubernetes集群
建议：Kubernetes版本需要与上面安装Kubernetes-cni依赖kubelet的版本相同！
[root@minikube ~]# minikube start --vm-driver=none --cni=flannel --image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers'
乐在分享！~~
