kubernetes系列-高可用k8s集群部署
Kubernetes高可用部署参考：
https://kubernetes.io/docs/setup/independent/high-availability/ https://github.com/kubernetes-sigs/kubespray https://github.com/wise2c-devops/breeze https://github.com/cookeem/kubeadm-ha 本次部署版本：
Kubernetes版本: 1.19.4 系统版本: CentOS7
一、环境准备
1.1 服务器规划
准备4台2C2G的服务器进行，自己规划3台master一台slave
1.2 环境准备
1. 保证所有节点接入互联网并配置好YUM源（略）
2. 关闭防火墙，selinux（略）
3. 设置好主机名，做好解析、互信（略）
4. 配置好时间同步
yum -y install chrony
更改配置文件 启动（略）
chronyc sources
5. 关闭swap
# swapoff -a
# sed -i 's/.*swap/#&/' /etc/fstab
6. 配置内核参数# vim /etc/sysctl.d/kubernetes.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 vm.swappiness=0
# sysctl -p /etc/sysctl.d/kubernetes.conf# 执行可能会出现一些错误，不鸟他
7. 加载ipvs模块# vim /etc/sysconfig/modules/ipvs.modules #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 # chmod +x /etc/sysconfig/modules/ipvs.modules # /etc/sysconfig/modules/ipvs.modules 8. 升级内核（版本较新的不需要此步） # yum -y update kernel # shutdown -r now
二、部署过程
1. 安装配置docker # 所有节点
2. 安装软件 # 所有节点
3. 安装负载均衡及高可用 # 所有 Master节点
4. 初台化Master1 # Master1节点
5. 配置kubectl # 所有需要的节点
6. 部署网络插件 # Master1节点
7. 加入Master节点 # 其它 Master节点
8. 加入Worker节点 # 所有 Node节点2.1 安装配置docker
在所有节点
安装步骤（略）
配置docker
# mkdir /etc/docker # vim /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "registry-mirrors": ["https://pf5f57i3.mirror.aliyuncs.com"] }
启动docker（略）
2.2 安装软件
所有节点
配置yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
安装软件
yum install -y kubeadm-1.19.4-0 kubelet-1.19.4-0 kubectl-1.19.4-0 ipvsadm
设置开机启动kubelet
systemctl enable kubelet
2.3 安装负载均衡
所有master节点操作
kubernetes master 节点运行如下主件：
- kube-apiserver
- kube-scheduler
- kube-controller-managerkube-scheduler 和 kube-controller-manager 可以以集群模式运行，通过 leader 选举产生一个工作进程，其它进程处于阻塞模式。kube-apiserver可以运行多个实例，但对其它组件需要提供统一的访问地址，该地址需要高可用。
本次部署使用 keepalived+haproxy 实现 kube-apiserver 的VIP 高可用和负载均衡。keepalived 提供 kube-apiserver 对外服务的 VIP高可用。haproxy 监听 VIP，后端连接所有 kube-apiserver 实例，提供健康检查和负载均衡功能。kube-apiserver的端口为6443, 为避免冲突, haproxy 监听的端口要与之不同，此实验中为6444。
keepalived 周期性检查本机的 haproxy 进程状态，如果检测到 haproxy 进程异常，则触发VIP 飘移。 所有组件都通过 VIP 监听的6444端口访问 kube-apiserver 服务。
在此我们使用睿云智合相关镜像，具体使用方法请访问： https://github.com/wise2c-devops 。当然也可以手动配置haproxy以实现apiserver负载均衡，keepalived实现haproxy高可用。
创建 haproxy和 keepalived的启动脚本
# vim haproxy.sh #!/bin/bash MasterIP1=192.168.10.11 MasterIP2=192.168.10.12 MasterIP3=192.168.10.13 MasterPort=6443 # apiserver端口 docker run -d --restart=always --name haproxy-k8s -p 6444:6444 \ -e MasterIP1=$MasterIP1 \ -e MasterIP2=$MasterIP2 \ -e MasterIP3=$MasterIP3 \ -e MasterPort=$MasterPort wise2c/haproxy-k8s ----------------------------------------------------------------- # vim keepalived.sh #!/bin/bash VIRTUAL_IP=192.168.10.100 # VIP INTERFACE=ens33 # 网卡名称 NETMASK_BIT=24 CHECK_PORT=6444 # Haproxy端口 RID=10 VRID=160 MCAST_GROUP=224.0.0.18 docker run -itd --restart=always --name=keepalived-k8s \ --net=host --cap-add=NET_ADMIN \ -e VIRTUAL_IP=$VIRTUAL_IP \ -e INTERFACE=$INTERFACE \ -e NETMASK_BIT=$NETMASK_BIT \ -e CHECK_PORT=$CHECK_PORT \ -e RID=$RID -e VRID=$VRID \ -e MCAST_GROUP=$MCAST_GROUP wise2c/keepalived-k8s
在所有Master节点执行脚本运行这两个容器
# sh haproxy.sh
# sh keepalived.sh
测试
1).在每台机器上查看容器(haproxy, keepalived)是否都正常运行
2).在每台机器上查看6444端口是否监听
3).在有VIP的机器关闭haproxy容器或keepalived容器看看VIP能否正常飘移2.4 初始化master1
仅在master1上操作
参考文档：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
在master1上创建初始化配置文件
[root@master1 k8s]# vim init.yml apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.10.11 # 此处改为本机IP bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: master1 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "192.168.10.100:6444" # VIP：PORT controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers # 使用国内镜像仓库 kind: ClusterConfiguration kubernetesVersion: v1.18.0 # 版本号 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 # pod子网，和Flannel中要一致 scheduler: {} --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvs 初始化master1 [root@master1 k8s]# kubeadm init --config=init.yml --upload-certs |tee kubeadm-init.log W0330 13:01:44.434476 4560 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.0 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.11 192.168.10.100] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master1 localhost] and IPs [192.168.10.11 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master1 localhost] and IPs [192.168.10.11 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" W0330 13:01:50.324799 4560 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0330 13:01:50.329031 4560 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 39.597752 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster [kubelet-check] Initial timeout of 40s passed. [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: b696a3afc89a8c60e130028d66be172c348ee80c789fcec6f79f759142eea6b8 [mark-control-plane] Marking the node master1 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.10.100:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:673d71fd341c79d3a013993c546bbf529f8626506f8d14fc69f0be376956e56f \ --control-plane --certificate-key b696a3afc89a8c60e130028d66be172c348ee80c789fcec6f79f759142eea6b8 Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.100:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:673d71fd341c79d3a013993c546bbf529f8626506f8d14fc69f0be376956e56f
kubeadm init主要执行了以下操作： [init]：指定版本进行初始化操作 [preflight] ：初始化前的检查和下载所需要的Docker镜像文件 [kubelet-start]：生成kubelet的配置文件”"var/lib/kubelet/config.yaml"，没有这个文件kubelet无法启动，所以初始化之前的kubelet实际上启动不会成功。 [certificates]：生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。 [kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目录中，组件之间通信需要使用对应文件。 [control-plane]：使用/etc/kubernetes/manifest目录下的YAML文件，安装 Master 组件。 [etcd]：使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。 [wait-control-plane]：等待control-plan部署的Master组件启动。 [apiclient]：检查Master组件服务状态。 [uploadconfig]：更新配置 [kubelet]：使用configMap配置kubelet。 [patchnode]：更新CNI信息到Node上，通过注释的方式记录。 [mark-control-plane]：为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。 [bootstrap-token]：生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到 [addons]：安装附加组件CoreDNS和kube-proxy
2.5 配置kubectl
无论在master节点或node节点，要能够执行kubectl命令必须进行配置，有两种配置方式
方式一，通过配置文件
[root@master k8s]# mkdir -p $HOME/.kube [root@master k8s]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master k8s]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
方式二，通过环境变量
# echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >> ~/.bashrc # source ~/.bashrc
配置好kubectl后，就可以使用kubectl命令了
[root@master1 k8s]# kubectl get cs （ps：会报错不要紧，下面有解决办法） Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"}
[root@master1 k8s]# kubectl get no
NAME STATUS ROLES AGE VERSION
master1 NotReady master 2m32s v1.19.4
[root@master1 k8s]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-965nt 0/1 Pending 0 2m24s
coredns-7ff77c879f-qmsb2 0/1 Pending 0 2m24s
etcd-master1 1/1 Running 1 2m38s
kube-apiserver-master1 1/1 Running 1 2m38s
kube-controller-manager-master1 1/1 Running 1 2m38s
kube-proxy-q847x 1/1 Running 1 2m24s
kube-scheduler-master1 1/1 Running 1 2m38s由于未安装网络插件，coredns处于pending状态，node处于NotReady 状态
在更高版本的k8s集群中，可能会出现 scheduler、controller-manager 两个组件健康状况不正常的情况，如下： [root@master-1 k8s]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused etcd-0 Healthy {"health":"true"} 解决方法：修改以下两个文件： /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-scheduler.yaml 注释掉： - --port=0 ，等一会即可
2.6 部署网络插件
仅在Master1节点操作
kubernetes支持多种网络方案，这里简单介绍常用的 flannel 方案.
下载kube-flannel.yml文件
[root@master k8s]# curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
由于kube-flannel.yml文件指定的镜像从coreos镜像仓库拉取，可能拉取失败，我们替换掉它
# 查看[root@master k8s]# grep -i "flannel:" kube-flannel.yml image: quay.io/coreos/flannel:v0.12.0-amd64 image: quay.io/coreos/flannel:v0.12.0-amd64
# 替换（注：因文件版本不同，替换命令可能有所不同）
[root@master k8s]# sed -i 's#quay.io/coreos/flannel:v0.12.0-amd64#registry.cn-shenzhen.aliyuncs.com/jun-lin/flannel:v0.13#' kube-flannel.yml执行安装
[root@master1 k8s]# kubectl apply -f kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created
再次查看node和 Pod状态，全部OK, 所有的核心组件都起来了
[root@master1 k8s]# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-965nt 1/1 Running 0 24m coredns-7ff77c879f-qmsb2 1/1 Running 0 24m etcd-master1 1/1 Running 1 25m kube-apiserver-master1 1/1 Running 1 25m kube-controller-manager-master1 1/1 Running 1 25m kube-flannel-ds-amd64-vvj65 1/1 Running 0 48s kube-proxy-q847x 1/1 Running 1 24m kube-scheduler-master1 1/1 Running 1 25m [root@master1 k8s]# kubectl get no NAME STATUS ROLES AGE VERSION master1 Ready master 26m v1.19.4
2.7 加入master节点
在另外两个master节点执行
[root@master2 ~]# kubeadm join 192.168.10.100:6444 --token abcdef.0123456789abcdef \ > --discovery-token-ca-cert-hash sha256:673d71fd341c79d3a013993c546bbf529f8626506f8d14fc69f0be376956e56f \ > --control-plane --certificate-key b696a3afc89a8c60e130028d66be172c348ee80c789fcec6f79f759142eea6b8 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [preflight] Running pre-flight checks before initializing the new control plane instance [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master2 localhost] and IPs [192.168.10.12 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master2 localhost] and IPs [192.168.10.12 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.12 192.168.10.100] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Valid certificates and keys now exist in "/etc/kubernetes/pki" [certs] Using the existing "sa" key [kubeconfig] Generating kubeconfig files [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" W0330 13:04:03.664147 5524 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0330 13:04:03.683264 5524 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0330 13:04:03.687046 5524 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [check-etcd] Checking that the etcd cluster is healthy [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [etcd] Announced new etcd member joining to the existing etcd cluster [etcd] Creating static Pod manifest for "etcd" [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s [kubelet-check] Initial timeout of 40s passed. [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [mark-control-plane] Marking the node master2 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane (master) label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster.
再查看集群信息
[root@master1 k8s]# kubectl get no NAME STATUS ROLES AGE VERSION master1 Ready master 88m v1.19.4 master2 Ready master 53m v1.19.4 master3 Ready master 48m v1.19.4 [root@master1 k8s]# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE coredns-7ff77c879f-r4mqp 1/1 Running 0 12m coredns-7ff77c879f-w9nsh 1/1 Running 0 12m etcd-master1 1/1 Running 0 13m etcd-master2 1/1 Running 0 9m39s etcd-master3 1/1 Running 0 3m51s kube-apiserver-master1 1/1 Running 0 13m kube-apiserver-master2 1/1 Running 0 11m kube-apiserver-master3 1/1 Running 0 3m50s kube-controller-manager-master1 1/1 Running 1 13m kube-controller-manager-master2 1/1 Running 1 9m43s kube-controller-manager-master3 1/1 Running 0 3m50s kube-flannel-ds-amd64-9wn57 1/1 Running 1 11m kube-flannel-ds-amd64-ktxpl 1/1 Running 0 12m kube-flannel-ds-amd64-qhttx 1/1 Running 0 8m59s kube-proxy-6hlql 1/1 Running 0 11m kube-proxy-jbx8r 1/1 Running 0 12m kube-proxy-l9782 1/1 Running 0 8m59s kube-scheduler-master1 1/1 Running 2 13m kube-scheduler-master2 1/1 Running 2 9m43s kube-scheduler-master3 1/1 Running 0 3m49s
2.8 加入node节点
在所有node节点执行
[root@node1 ~]# kubeadm join 192.168.10.100:6444 --token abcdef.0123456789abcdef \ > --discovery-token-ca-cert-hash sha256:673d71fd341c79d3a013993c546bbf529f8626506f8d14fc69f0be376956e56f W0330 13:19:50.191449 3623 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
过一段时间再看集群信息
[root@master1 k8s]# kubectl get no NAME STATUS ROLES AGE VERSION master1 Ready master 88m v1.19.4 master2 Ready master 53m v1.19.4 master3 Ready master 48m v1.19.4 node1 Ready <none> 34m v1.19.4 至此，集群部署完毕。
三、问题解决
如果安装过程中出现问题, 无论是Master还是Node, 都可以执行 kubeadm reset 命令进行重置
[root@node2 ~]# kubeadm reset [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted. [reset] Are you sure you want to proceed? [y/N]: y [preflight] Running pre-flight checks W0321 22:54:01.292739 7918 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory [reset] No etcd config found. Assuming external etcd [reset] Please, manually reset etcd to prevent further issues [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] [reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] # reset进程没有清理CNI配置 The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d # reset进程没有清理iptables或IPVS表 The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables. # reset进程没有清理kubeconfig文件 The reset process does not clean your kubeconfig files and you must remove them manually. Please, check the contents of the $HOME/.kube/config file.
