基于 KubeSphere 的 Kubernetes 管理平台最佳实践

简介

　　KubeSphere 是在 Kubernetes 之上构建的面向云原生应用的分布式操作系统，完全开源，支持多云与多集群管理，提供全栈的 IT 自动化运维能力，简化企业的 DevOps 工作流。它的架构可以非常方便地使第三方应用与云原生生态组件进行即插即用 (plug-and-play) 的集成。
　　KubeSphere 还开源了 KubeKey 帮助企业一键在公有云或数据中心快速搭建 Kubernetes 集群，提供单节点、多节点、集群插件安装，以及集群升级与运维。

一、KubeKey和KubeSphere

?

1 2 3

工具介绍： https://github.com/kubesphere/kubekey/blob/master/README_zh-CN.md https://github.com/kubesphere/kubesphere/blob/master/README_zh.md

二、环境准备

　　1、三台8核/16G内存/500G高速硬盘服务器和三台16核/32G内存/500G高速硬盘服务器

　　2、CentOS 7

　　3、关闭selinux和关闭防火墙

　　4、升级系统内核（参考：https://www.cnblogs.com/a120608yby/p/10169364.html）

　　

三、依赖安装

?

1	yum install socat conntrack ebtables ipset -y

四、下载KubeKey

?

1 2 3 4

export KKZONE=cn curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.10 sh - chmod +x kk ./kk create config --with-kubesphere v3.4.0 --with-kubernetes v1.23.10

五、编辑config-sample.yaml

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252

apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata:   name: sample spec:   hosts:   - {name: k8s-master1, address: 172.16.0.1, internalAddress: 172.16.0.1, user: root, password: "Qcloud@123"}   - {name: k8s-master2, address: 172.16.0.2, internalAddress: 172.16.0.2, user: root, password: "Qcloud@123"}   - {name: k8s-master3, address: 172.16.0.3, internalAddress: 172.16.0.3, user: root, password: "Qcloud@123"}   - {name: k8s-node1, address: 172.16.0.4, internalAddress: 172.16.0.4, user: root, password: "Qcloud@123"}   - {name: k8s-node2, address: 172.16.0.5, internalAddress: 172.16.0.5, user: root, password: "Qcloud@123"}   - {name: k8s-node3, address: 172.16.0.6, internalAddress: 172.16.0.6, user: root, password: "Qcloud@123"}   roleGroups:     etcd:     - k8s-master[1:3]     control-plane:     - k8s-master[1:3]     worker:     - k8s-node[1:3]   controlPlaneEndpoint:     ## Internal loadbalancer for apiservers     internalLoadbalancer: haproxy       domain: lb.kubesphere.local     address: ""     port: 6443   kubernetes:     version: v1.23.10     clusterName: cluster.local     autoRenewCerts: true     containerManager: docker   etcd:     type: kubekey   network:     plugin: calico     kubePodsCIDR: 10.233.64.0/18     kubeServiceCIDR: 10.233.0.0/18     ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni     multusCNI:       enabled: false   registry:     privateRegistry: ""     namespaceOverride: ""     registryMirrors: ["https://docker.m.daocloud.io", "https://dockerproxy.com", "https://docker.mirrors.ustc.edu.cn", "https://docker.nju.edu.cn"]     insecureRegistries: ["harbor.example.com"]   addons: []       --- apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata:   name: ks-installer   namespace: kubesphere-system   labels:     version: v3.4.0 spec:   persistence:     storageClass: ""   authentication:     jwtSecret: ""   zone: ""   local_registry: ""   namespace_override: ""   # dev_tag: ""   etcd:     monitoring: true     endpointIps: localhost     port: 2379     tlsEnable: true   common:     core:       console:         enableMultiLogin: true         port: 30880         type: NodePort     # apiserver:     #  resources: {}     # controllerManager:     #  resources: {}     redis:       enabled: false       enableHA: false       volumeSize: 2Gi     openldap:       enabled: false       volumeSize: 2Gi     minio:       volumeSize: 20Gi     monitoring:       # type: external       endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090       GPUMonitoring:         enabled: false     gpu:       kinds:       - resourceName: "nvidia.com/gpu"         resourceType: "GPU"         default: true     es:       # master:       #   volumeSize: 4Gi       #   replicas: 1       #   resources: {}       # data:       #   volumeSize: 20Gi       #   replicas: 1       #   resources: {}       logMaxAge: 7       elkPrefix: logstash       basicAuth:         enabled: false         username: ""         password: ""       externalElasticsearchHost: ""       externalElasticsearchPort: ""     opensearch:       # master:       #   volumeSize: 4Gi       #   replicas: 1       #   resources: {}       # data:       #   volumeSize: 20Gi       #   replicas: 1       #   resources: {}       enabled: true       logMaxAge: 7       opensearchPrefix: whizard       basicAuth:         enabled: true         username: "admin"         password: "admin"       externalOpensearchHost: ""       externalOpensearchPort: ""       dashboard:         enabled: false   alerting:     enabled: true     # thanosruler:     #   replicas: 1     #   resources: {}   auditing:     enabled: true     # operator:     #   resources: {}     # webhook:     #   resources: {}   devops:     enabled: false     jenkinsCpuReq: 0.5     jenkinsCpuLim: 1     jenkinsMemoryReq: 4Gi     jenkinsMemoryLim: 4Gi     jenkinsVolumeSize: 16Gi   events:     enabled: true     # operator:     #   resources: {}     # exporter:     #   resources: {}     # ruler:     #   enabled: true     #   replicas: 2     #   resources: {}   logging:     enabled: true     logsidecar:       enabled: true       replicas: 2       # resources: {}   metrics_server:     enabled: true   monitoring:     storageClass: ""     node_exporter:       port: 9100       # resources: {}     # kube_rbac_proxy:     #   resources: {}     # kube_state_metrics:     #   resources: {}     # prometheus:     #   replicas: 1     #   volumeSize: 20Gi     #   resources: {}     #   operator:     #     resources: {}     # alertmanager:     #   replicas: 1     #   resources: {}     # notification_manager:     #   resources: {}     #   operator:     #     resources: {}     #   proxy:     #     resources: {}     gpu:       nvidia_dcgm_exporter:         enabled: false         # resources: {}   multicluster:     clusterRole: none   network:     networkpolicy:       enabled: true     ippool:       type: calico     topology:       type: weave-scope   openpitrix:     store:       enabled: true   servicemesh:     enabled: true     istio:       components:         ingressGateways:         - name: istio-ingressgateway           enabled: false         cni:           enabled: false   edgeruntime:     enabled: true     kubeedge:       enabled: true       cloudCore:         cloudHub:           advertiseAddress:             - "172.16.0.1"         service:           cloudhubNodePort: "30000"           cloudhubQuicNodePort: "30001"           cloudhubHttpsNodePort: "30002"           cloudstreamNodePort: "30003"           tunnelNodePort: "30004"         # resources: {}         # hostNetWork: false       iptables-manager:         enabled: true         mode: "external"         # resources: {}       # edgeService:       #   resources: {}   gatekeeper:     enabled: true     # controller_manager:     #   resources: {}     # audit:     #   resources: {}   terminal:     timeout: 600

六、部署 KubeSphere 和 Kubernetes

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

./kk create cluster -f config-sample.yaml kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f   # 看到以下日志，证明部署成功 ##################################################### ###              Welcome to KubeSphere!           ### #####################################################      Console: http://172.16.0.1:30880 Account: admin Password: P@88w0rd      NOTES：   1. After you log into the console, please check the      monitoring status of service components in      the "Cluster Management". If any service is not      ready, please wait patiently until all components      are up and running.   2. Please change the default password after login.      ##################################################### https://kubesphere.io             2022-xx-xx xx:xx:xx #####################################################

七、命令自动补全

?

1 2 3 4 5

yum -y install bash-completion   echo 'source <(kubectl completion bash)' >> ~/.bashrc 或者 kubectl completion bash > /etc/bash_completion.d/kubectl

参考：

?

1	https://kubesphere.io/zh/docs/v3.3/installing-on-linux/high-availability-configurations/internal-ha-configuration/