kubeasz搭建k8s高可用集群

kubeasz致力于快速部署高可用k8s集群的工具，同时也努力成为k8s实践、使用的参考书；基于二进制方式部署和ansible-playbook实现自动化；既提供一键安装脚本，可以根据安装指南分步执行安装各个组件。

Kubeasz从每一个单独部件组装到完整的集群，提供最灵活的配置能力，几乎可以设置任何组件的任何参数化；同时又为集群创建预置一套运行良好的默认配置。

节点规划：均是基于centos7.9镜像，2G1cpu30G

角色	服务器	描述
部署节点	192.168.238.11	作为宿主机通过kubeasz容器运行ansible/ezctl命令
etcd节点	192.168.238.12 192.168.238.13 192.168.238.14	注意etcd集群需要1,3,5,...奇数个节点，本实战安装3个节点
master节点	192.168.238.12 192.168.238.13 192.168.238.14	高可用集群至少2个master节点，本实战安装3个节点
node节点	192.168.238.15	运行应用负载的节点，节点数任意，本实战安装1个节点

kubeasz使用3.3.0版本，搭建k8s集群信息如下：

k8s: v1.24.1

docker:v20.10.16

ansible:v2.10.8

etcd: v3.5.4

containerd: 1.6.4

flanal: v0.15.1

dashboard: v2.5.1

以下所有操作均在部署节点完成：

设置免密登录：

ssh-keygen
ssh-copy-id 192.168.238.12
ssh-copy-id 192.168.238.13
ssh-copy-id 192.168.238.14
ssh-copy-id 192.168.238.15

准备环境：

# 下载工具脚本ezdown，使用kubeasz版本3.3.0
export release=3.3.0
yum install wget -y
wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown
chmod +x ./ezdown
# 使用工具脚本下载k8s需要的依赖和镜像
./ezdown -D
#./ezdown -D命令多执行几遍，直至再执行的时候不再downloading为止。这样就在/etc/kubeasz目录下下载了在线安装所有需要的脚本、二进制文件和镜像文件。
./ezdown -P （可选）

如果目标服务器可以直接连外网，那么通过./ezdown -D下载的文件足够用了，但是如果目标服务器不能连外网，那么我们还需要通过另外一个命令./ezdown -P把离线文件也下载下来，这些文件也都在/etc/kubeasz。后面我们把/etc/kubeasz这个目录打个压缩包，并和ezdown文件归档在一起后续可以直接使用。

/etc/kubeasz 包含 kubeasz 版本为 ${release} 的发布代码。
/etc/kubeasz/bin 包含 kubernetes/etcd/docker/cni 等二进制文件。
/etc/kubeasz/down 包含集群安装时需要的离线容器镜像。
/etc/kubeasz/down/packages 包含集群安装时需要的系统基础软件。

容器化运行kubeasz：

./ezdown -S
2023-11-07 09:25:58 INFO Action begin: start_kubeasz_docker
2023-11-07 09:25:58 INFO try to run kubeasz in a container
2023-11-07 09:25:58 DEBUG get host IP: 192.168.238.11
Loaded image: easzlab/kubeasz:3.3.0
de281307dce0ce3e27c8ec28bfc5ad3bd60b700351dd4d0827aeeb4c3ac6ba41
2023-11-07 09:26:00 INFO Action successed: start_kubeasz_docker

进入kubeasz创建集群k8s-01:

docker exec -it kubeasz /bin/bash  #进入容器
bash-5.1# ezctl new k8s-01    #创建k8s-01集群
2023-11-07 01:26:51 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-01
2023-11-07 01:26:51 DEBUG set versions
2023-11-07 01:26:51 DEBUG cluster k8s-01: files successfully created.
2023-11-07 01:26:51 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-01/hosts'
2023-11-07 01:26:51 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-01/config.yml'

集群信息确定：

修改/etc/kubeasz/clusters/k8s-01/hosts的文件信息：

# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
192.168.238.12
192.168.238.13
192.168.238.14

# master node(s)
[kube_master]
192.168.238.12
192.168.238.13
192.168.238.14

# work node(s)
[kube_node]
192.168.238.15

# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false

# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
#192.168.1.1

[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"

# Cluster container-runtime supported: docker, containerd
# if k8s version >= 1.24, docker is not supported
CONTAINER_RUNTIME="containerd"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="flannel"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="172.20.0.0/16"

# NodePort Range
NODE_PORT_RANGE="30000-32767"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local"

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"

# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-01"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

修改hosts，config.yml保持不变。其中hosts文件中按规划调整了etcd、kube_master、kube_node和ex_lb四处位置的服务器IP，注意这里只能使用IP，不能使用hostname；另外CONTAINER_RUNTIME应该设置为containerd，CLUSTER_NETWORK设置为flannel，其它配置可保持不变。

一键安装：

docker exec -it kubeasz /bin/bash
bash-5.1# ezctl setup k8s-01 all

验证安装：

#可以看到各节点就绪 (Ready) 状态、角色、运行时间以及版本号
bash-5.1# kubectl get nodes
NAME             STATUS                     ROLES    AGE   VERSION
192.168.238.12   Ready,SchedulingDisabled   master   9h    v1.24.1
192.168.238.13   Ready,SchedulingDisabled   master   9h    v1.24.1
192.168.238.14   Ready,SchedulingDisabled   master   9h    v1.24.1
192.168.238.15   Ready                      node     9h    v1.24.1
# 可以看到scheduler/controller-manager/etcd等组件 Healthy
bash-5.1# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
etcd-2               Healthy   {"health":"true","reason":""}
etcd-1               Healthy   {"health":"true","reason":""}
etcd-0               Healthy   {"health":"true","reason":""}
controller-manager   Healthy   ok
scheduler            Healthy   ok
# 可以看到kubernetes master(apiserver)组件 running
bash-5.1# kubectl cluster-info
Kubernetes control plane is running at https://192.168.238.12:6443
CoreDNS is running at https://192.168.238.12:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
KubeDNSUpstream is running at https://192.168.238.12:6443/api/v1/namespaces/kube-system/services/kube-dns-upstream:dns/proxy
kubernetes-dashboard is running at https://192.168.238.12:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
# 可以查看所有集群pod状态，默认已安装网络插件flannel、coredns、metrics-server等
bash-5.1# kubectl get po --all-namespaces
NAMESPACE     NAME                                        READY   STATUS    RESTARTS        AGE
kube-system   coredns-ff4774677-xxsxv                     1/1     Running   1               9h
kube-system   dashboard-metrics-scraper-8c47d4b5d-j256c   1/1     Running   0               9h
kube-system   kube-flannel-ds-7llfk                       1/1     Running   0               9h
kube-system   kube-flannel-ds-fvm45                       1/1     Running   0               9h
kube-system   kube-flannel-ds-wzhvx                       1/1     Running   0               9h
kube-system   kube-flannel-ds-xc69z                       1/1     Running   0               9h
kube-system   kubernetes-dashboard-5d46f4c997-jwj6z       1/1     Running   7 (7h12m ago)   9h
kube-system   metrics-server-56646b5b79-9lq46             1/1     Running   7 (7h13m ago)   9h
kube-system   node-local-dns-c2cfs                        1/1     Running   2               9h
kube-system   node-local-dns-hp5nc                        1/1     Running   2 (7h16m ago)   9h
kube-system   node-local-dns-tcvzb                        1/1     Running   0               9h
kube-system   node-local-dns-vjtfn                        1/1     Running   0               9h
# 可以查看所有集群svc状态
bash-5.1# kubectl get svc --all-namespaces
NAMESPACE     NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes                  ClusterIP   10.68.0.1       <none>        443/TCP                  9h
kube-system   dashboard-metrics-scraper   ClusterIP   10.68.79.205    <none>        8000/TCP                 9h
kube-system   kube-dns                    ClusterIP   10.68.0.2       <none>        53/UDP,53/TCP,9153/TCP   9h
kube-system   kube-dns-upstream           ClusterIP   10.68.129.173   <none>        53/UDP,53/TCP            9h
kube-system   kubernetes-dashboard        NodePort    10.68.119.173   <none>        443:30974/TCP            9h
kube-system   metrics-server              ClusterIP   10.68.81.213    <none>        443/TCP                  9h
kube-system   node-local-dns              ClusterIP   None            <none>        9253/TCP                 9h

登录dashboard界面：

https://节点任意ip地址:30974

从master节点查看token并使用token登陆（这里为了方便，我们可以直接使用admin-user的token）
# 查看内容含有token的secret
bash-5.1# kubectl get secret -n kube-system
NAME                              TYPE                                  DATA   AGE
admin-user                        kubernetes.io/service-account-token   3      9h
dashboard-read-user               kubernetes.io/service-account-token   3      9h
kubernetes-dashboard-certs        Opaque                                0      9h
kubernetes-dashboard-csrf         Opaque                                1      9h
kubernetes-dashboard-key-holder   Opaque                                2      9h
# 查看admin-user对应的token
bash-5.1# kubectl describe secret -n kube-system admin-user
Name:         admin-user
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: 10b7016f-8ac0-44e0-b24b-6743f09914ee

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1302 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImhTLWxIQ1FSSmpfalZaNW5WTDZUcHI2TWYwUUhOcnlYUzB2aXFMeGt4eVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxMGI3MDE2Zi04YWMwLTQ0ZTAtYjI0Yi02NzQzZjA5OTE0ZWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.gEb8k2haf4S9-KMImjoTwPqt5-iBuVeKiuVwUE6Vav0Dmtx-_oo4EZioNxFaoBXcW5GswUqSF_S7PM1V5Y3EW__Qsx-xK2ado20Q4oGqwDDYM1b6NaqgMdvwfZuxdNH-ydYot9BRBjOETDu-UgQJ1b_KIHKU5niTX9htS6UohxqArZbgVKviBs5a0dz4Cx5-tw13W6IzeRd-zeVn38ed_YeW3zPLdNE0QnnorqXATxpekAWcbKNy2dmxckTR3990ZVCQ9u8FY4_AebcVEQWUvuCnEaEUBxQLQ_tKFAJ6PVe9UTxsJGZoT1RI9NQvkwLvKYRxaWjoy_VqZuW4KWG1BA
bash-5.1#