KubeKey快速构建Kubernetes KubeSphere集群
使用KubeKey快速构建Kubernetes、KubeSphere集群
安装须知
下图是根据KubeKey官网所给出的一些安装建议和要求，构建前请仔细阅读
环境准备
通过阅读前面KubeKey所给出的建议，相信您已经了解构建整个Kubernetes以及kubeSphere之前的环境准备。下面是对其环境准备前的部分简单总结
-
官方推荐机器要求最低配置为（2核：CPU+4G：运行内存+20G：磁盘存储），为了更好的使用KubeSphere的体验感，最好配置为双倍的资源。
-
对于网络，整个集群之间的
网络能互相通讯，且DNS能正常工作，每台主机之间都能使用SSH来登录主机，且每台主机能正常访问互联网 -
对于时间，所有节点的时间都需要
同步为一致的时间 -
对于Docker镜像加速，可参考后续给出的配置文件
-
更多的环境准备可查看上图或 KubeKey官网 .........
根据以上的配置要求及其环境准备对整个的环境准备构建了如下的
shell执行脚本读者，可根据需求自取。
################## 以下命令所有机器都需要执行 ##################################
# 安装基础组件
yum install -y wget vim yum-utils socat conntrack ebtables ipset ipvsadm
# 备份源
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
# 切换阿里云镜像源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# 更新源缓存
yum makecache
# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
# 关闭网络管理
systemctl stop NetworkManager
systemctl disable NetworkManager
# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 临时
# 设置时区
timedatectl set-timezone Asia/Shanghai
# 重启chronyd服务并设为开机启动：
systemctl enable chronyd && systemctl restart chronyd
# 卸载老版本的Docker（非必须）
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
# 设置安装 docker 的 yum源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 更新缓存
yum makecache
# 安装docker
yum install -y docker-ce docker-ce-cli containerd.io
# docker 启动 设置开机自启
systemctl enable docker && systemctl start docker
# 配置下载镜像加速源并修改组
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://1230ozco.mirror.aliyuncs.com"]
}
EOF
# 重启
sudo systemctl daemon-reload
sudo systemctl restart docker
############ 如果使用Centos7 那么内核版本可能是3.x版本的内核，推荐升级内核，因为3.x版本内核会导致Kubernetes运行不稳定 ############
# 如果系统内核以已经升级请忽略
# 修改系统内核
yum -y update
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
yum --enablerepo=elrepo-kernel install kernel-ml -y
sudo awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
# 重启系统
reboot
开始安装
通过上面都操作，你将获得一个可安装kubernetes和kubesphere环境的集群，接下来将在已经准备好的机器环境下进行操作来构建集群
使用KubeKey来构建集群，将为我们省去非常繁琐的构建集群方式，它将变得更简单，集体只需要如下几步便可构建一个高可用的Kubernetes和KuberSphere集群。
-
下载KubeKey
-
编辑集群配置文件
-
根据配置文件创建集群即可
根据以上步骤我们构建一个基于
Kubernetes和KubeSphere的集群只需要执行如下的shell脚本即可。
### 前面通过配置了ssh 以下操作只需要在构建集群的任意一台机器上执行即可（最好还是推荐在master上执行，方便未来使用KubeKey管理集群升级操作） ###
#下载KubeKey
export KKZONE=cn
curl -sfL https://get-kk.kubesphere.io | VERSION=v1.2.1 sh -
tar -xzvf kubekey-v1.2.1-linux-amd64.tar.gz
mv kk /usr/local/bin/
rm -rf kubekey-v1.2.1-linux-amd64.tar.gz
rm -rf README.md
rm -rf README_zh-CN.md
# 参考版本
kk version
### 编辑配置文件 =》 对于配置文件，可能需要安装的kubernetes和kubesphere版本不同，构建的机器IP不同，选择启用的功能不同，可能会有所差异 ####
### 对于这部分的差异可参考后续的配置文件详解来根据需求来自定义修改 ###
cat > config.yaml <<EOF
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: master-node1, address: 192.168.20.111, internalAddress: 192.168.20.111, user: root, password: '970699'}
- {name: master-node2, address: 192.168.20.112, internalAddress: 192.168.20.112, user: root, password: '970699'}
- {name: work-node1, address: 192.168.20.113, internalAddress: 192.168.20.113, user: root, password: '970699'}
- {name: work-node2, address: 192.168.20.114, internalAddress: 192.168.20.114, user: root, password: '970699'}
roleGroups:
etcd:
- master-node1 # etcd 个数只能是奇数
master:
- master-node1
- master-node2
worker:
- work-node1
- work-node2
controlPlaneEndpoint:
##Internal loadbalancer for apiservers
internalLoadbalancer: haproxy
domain: lb.kubesphere.local
address: ""
port: 6443
kubernetes:
version: v1.21.5
clusterName: cluster.local
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
registry:
registryMirrors: []
insecureRegistries: []
addons: []
---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.2.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
local_registry: ""
# dev_tag: ""
etcd:
monitoring: false
endpointIps: localhost
port: 2379
tlsEnable: true
common:
core:
console:
enableMultiLogin: true
port: 30880
type: NodePort
# apiserver:
# resources: {}
# controllerManager:
# resources: {}
redis:
enabled: false
volumeSize: 2Gi
openldap:
enabled: false
volumeSize: 2Gi
minio:
volumeSize: 20Gi
monitoring:
# type: external
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
GPUMonitoring:
enabled: false
gpu:
kinds:
- resourceName: "nvidia.com/gpu"
resourceType: "GPU"
default: true
es:
# master:
# volumeSize: 4Gi
# replicas: 1
# resources: {}
# data:
# volumeSize: 20Gi
# replicas: 1
# resources: {}
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchHost: ""
externalElasticsearchPort: ""
alerting:
enabled: false
# thanosruler:
# replicas: 1
# resources: {}
auditing:
enabled: false
# operator:
# resources: {}
# webhook:
# resources: {}
devops:
enabled: true
jenkinsMemoryLim: 1Gi
jenkinsMemoryReq: 750Mi
jenkinsVolumeSize: 4Gi
jenkinsJavaOpts_Xms: 256m
jenkinsJavaOpts_Xmx: 256m
jenkinsJavaOpts_MaxRAM: 1g
events:
enabled: false
# operator:
# resources: {}
# exporter:
# resources: {}
# ruler:
# enabled: true
# replicas: 2
# resources: {}
logging:
enabled: false
containerruntime: docker
logsidecar:
enabled: true
replicas: 2
# resources: {}
metrics_server:
enabled: false
monitoring:
storageClass: ""
# kube_rbac_proxy:
# resources: {}
# kube_state_metrics:
# resources: {}
# prometheus:
# replicas: 1
# volumeSize: 20Gi
# resources: {}
# operator:
# resources: {}
# adapter:
# resources: {}
# node_exporter:
# resources: {}
# alertmanager:
# replicas: 1
# resources: {}
# notification_manager:
# resources: {}
# operator:
# resources: {}
# proxy:
# resources: {}
gpu:
nvidia_dcgm_exporter:
enabled: false
# resources: {}
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: true
servicemesh:
enabled: false
kubeedge:
enabled: false
cloudCore:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
cloudhubPort: "10000"
cloudhubQuicPort: "10001"
cloudhubHttpsPort: "10002"
cloudstreamPort: "10003"
tunnelPort: "10004"
cloudHub:
advertiseAddress:
- ""
nodeLimit: "100"
service:
cloudhubNodePort: "30000"
cloudhubQuicNodePort: "30001"
cloudhubHttpsNodePort: "30002"
cloudstreamNodePort: "30003"
tunnelNodePort: "30004"
EOF
# 开始构建集群
kk create cluster -f config.yaml
安装结果展示
当你根据自己的需求修改配置文件后，并执行shell脚本后，你将会得到如下界面，代表集群已经构建完成了，下面就可以使用它了。
- 访问日志中所给出的控制台地址
http://192.168.20.10:30880 - 填写对应的账号
admin和密码P@88w0rd登录即可使用
通过以上简单的几个步骤就可以构建一个大规模的kubernetes和kubesphere集群，相信读者看到这里，最迷惑的还是集群的配置文件，下面我们将继续对整个集群的部署配置文件进行详细的解析。
集群配置文件详解
下面将对整个集群的配置文件进行补充说明，让读者能更大程度的构建自己的自定义集群
不难看出，以下的配置文件是上面的一部分，下面将通过这些配置文件来对整个集群配置做相关的解析
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: sample
spec:
hosts:
# 需要配置集群的主机
# - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, port: 8022, user: ubuntu, password: Testing123}
# name：主机名，address: ssh连接的IP支持公有私有 internalAddress: 私有IP port：ssh连接的端口默认22可省略 user：服务器用户名 password：服务器密码
# - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, privateKeyPath: "~/.ssh/id_rsa"} 也可使用此配置免密登录
- {name: master-node1, address: 192.168.20.111, internalAddress: 192.168.20.111, user: root, password: '970699'}
- {name: master-node2, address: 192.168.20.112, internalAddress: 192.168.20.112, user: root, password: '970699'}
- {name: work-node1, address: 192.168.20.113, internalAddress: 192.168.20.113, user: root, password: '970699'}
- {name: work-node2, address: 192.168.20.114, internalAddress: 192.168.20.114, user: root, password: '970699'}
# 主机权限分组
roleGroups:
# etcd 安装主机 只能是奇数个（方便选举）
etcd:
- master-node1
# master控制节点 在kubesphere3.3版本中 master 替换为了 control-plane 如下
# control-plane:
# - master-node1
# - master-node2
master:
- master-node1
- master-node2
# 工作节点
worker:
- work-node1
- work-node2
# 高可用节点负载均衡配置（内部和外部只能使用一种方式）
controlPlaneEndpoint:
# 启用内部负载均衡
internalLoadbalancer: haproxy
# 负载均衡内部访问域名
domain: lb.kubesphere.local
# 启用外部负载均衡 外部高可用负载均衡可参考 https://www.bookstack.cn/read/kubesphere-3.3-zh/d5c6b1e31842373c.md
# address: "192.168.20.vip" 虚拟IP
# port: 6443 k8s通讯的虚拟端口
kubernetes:
version: v1.21.5 # kubernetes版本
clusterName: cluster.local # 集群名称
network:
plugin: calico # 扁平化网络组件
kubePodsCIDR: 10.233.64.0/18 # pod IP分配地址
kubeServiceCIDR: 10.233.0.0/18 # service IP分配地址
registry:
registryMirrors: []
insecureRegistries: []
addons: []
########## 以下配置属于kubesphere的安装配置 可根据kubesphere 官方配置自定义调整 ##########
---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.2.1 # kubersphere版本
spec:
persistence:
storageClass: "" # 如果kubernetes没有默认指定StorageClass 这里需要自己指定
authentication:
jwtSecret: "" #保持授权的 jwtSecret 与主机集群一致。通过在主机集群上执行“kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret" 来检索 jwtSecret。
local_registry: "" # 添加本地注册地址
# dev_tag: "" # 安装kubesphere的镜像标签 默认与ks-install一致
etcd:
monitoring: false # 是否启用etcd监控 注意启用前需要创建一个Secret
endpointIps: localhost # etcd集群地址，可以是复数用,分割
port: 2379 # etcd 通讯端口
tlsEnable: true # 是否启用 https 通讯
common:
core:
console:
enableMultiLogin: true #是否启用多用户登录登录一个账号
port: 30880 # kubesphere web 控制台端口
type: NodePort
# apiserver:
# resources: {}
# controllerManager:
# resources: {}
redis:
enabled: false
volumeSize: 2Gi
openldap:
enabled: false
volumeSize: 2Gi
minio:
volumeSize: 20Gi
monitoring:
# type: external
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
GPUMonitoring:
enabled: false
gpu:
kinds:
- resourceName: "nvidia.com/gpu"
resourceType: "GPU"
default: true
es:
# master:
# volumeSize: 4Gi
# replicas: 1
# resources: {}
# data:
# volumeSize: 20Gi
# replicas: 1
# resources: {}
logMaxAge: 7 # 内置ES日志保留最长时间
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchHost: ""
externalElasticsearchPort: ""
alerting:
enabled: false # 启用KubeSphere 报警系统
# thanosruler:
# replicas: 1
# resources: {}
auditing:
enabled: false # 启用KubeSphere 审计功能
# operator:
# resources: {}
# webhook:
# resources: {}
devops:
enabled: true # 启用KubeSphere DevOps功能
jenkinsMemoryLim: 1Gi # DevOps资源配置
jenkinsMemoryReq: 750Mi
jenkinsVolumeSize: 4Gi
jenkinsJavaOpts_Xms: 256m
jenkinsJavaOpts_Xmx: 256m
jenkinsJavaOpts_MaxRAM: 1g
events:
enabled: false # 是否启用 KubeSphere 事件系统
# operator:
# resources: {}
# exporter:
# resources: {}
# ruler:
# enabled: true
# replicas: 2
# resources: {}
logging:
enabled: false # 是否开启KubeSphere日志功能
containerruntime: docker
logsidecar:
enabled: true
replicas: 2
# resources: {}
metrics_server:
enabled: false # 是否启用服务器检测插件 metricsServer
monitoring:
storageClass，: "" # 监控系统使用的是Prometheus 需要重新指定storageClass，默认使用k8s指定的storageClass
# kube_rbac_proxy:
# resources: {}
# kube_state_metrics:
# resources: {}
# prometheus:
# replicas: 1 # prometheus 副本数量
# volumeSize: 20Gi # prometheus 挂载卷大小
# resources: {}
# operator:
# resources: {}
# adapter:
# resources: {}
# node_exporter:
# resources: {}
# alertmanager:
# replicas: 1
# resources: {}
# notification_manager:
# resources: {}
# operator:
# resources: {}
# proxy:
# resources: {}
gpu:
nvidia_dcgm_exporter:
enabled: false # 是否启用GPU监控插件
# resources: {}
multicluster:
clusterRole: none # 设置集群角色 可选值 host（主机） | member（成员） | none（无角色）
network:
networkpolicy:
enabled: false # 是否启用Pod之间的网络隔离
ippool:
type: none # 是否使用组件启动Pod的IP池分配，none表示禁用，可选值为扁平化网络插件如 calico
topology:
type: none # 是否启用服务拓扑，none表示禁用，可选值为 weave-scope
openpitrix:
store:
enabled: true # 是否启用KubeSphere自带的应用商店
servicemesh:
enabled: false #是否开启服务网关（基于Istio）
kubeedge:
enabled: false # 是否开启kubeedge（边缘计算）
cloudCore:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
cloudhubPort: "10000"
cloudhubQuicPort: "10001"
cloudhubHttpsPort: "10002"
cloudstreamPort: "10003"
tunnelPort: "10004"
cloudHub:
advertiseAddress:
- ""
nodeLimit: "100"
service:
cloudhubNodePort: "30000"
cloudhubQuicNodePort: "30001"
cloudhubHttpsNodePort: "30002"
cloudstreamNodePort: "30003"
tunnelNodePort: "30004"
通过对上述查看以上的配置文件，信息读者已经大概能自定义自己的KuberSphere集群节点，那么下一步，我们将对KubeKey如何来管理集群的节点进行操作
KubeKey管理集群节点
-
添加节点
# 编辑创建集群的配置文件 将需要集群的集群和加入集群的权限配置完成 vim config.yaml # 配置如下部分 ··· spec: hosts: - {name: master1, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: [email protected]} - {name: node1, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: [email protected]} - {name: node2, address: 192.168.0.5, internalAddress: 192.168.0.5, user: root, password: [email protected]} roleGroups: etcd: - master1 control-plane: - master1 worker: - node1 - node2 ··· # 执行添加节点命令，完成即可 kk add nodes -f config.yaml -
删除节点
# 找到安装集群的配置文件 执行以下命令 nodeName为配置文件中需要删除的nodeName名称 kk delete node <nodeName> -f config-sample.yaml -
卸载 KubeSphere 和 Kubernetes
# 找到安装集群的配置文件 执行以下命令 kk delete cluster -f config.yaml
