kubeadm快速部署一个k8s集群

1. 前置知识点

1.1 生产环境可部署kubernetes集群的两种方式

目前生产部署Kubernetes集群主要有两种方式
- kuberadm
  
  Kubeadm是一个K8s部署工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。
- 二进制包
  
  从github下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。
  
  Kubeadm降低部署门槛，但屏蔽了很多细节，遇到问题很难排查。如果想更容易可控，推荐使用二进制包部署Kubernetes集群，虽然手动部署麻烦点，期间可以学习很多工作原理，也利于后期维护。

1.2 安装要求

在开始之前，部署Kubernetes集群机器需要满足以下几个条件：
- 一台或多台机器，操作系统 CentOS7.x-86_x64
- 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
- 集群中所有机器之间网络互通
- 可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点
- 禁止swap分区

1.3 环境准备

1.3.1 软件环境

软件	版本
操作系统	CentOS7.8_x64 （mini）
Docker	19-ce
Kubernetes	1.19

1.3.2 单master架构图

1.3.3 服务器整体规划

角色	IP	组件
k8s-master	192.168.0.201	kube-apiserver，kube-controller-manager，kube-scheduler，etcd
k8s-node1	192.168.0.202	kubelet，kube-proxy，docker etcd
k8s-node2	192.168.0.203	kubelet，kube-proxy，docker，etcd
镜像仓库	192.168.0.229	jenkins，harbor，docker，git，maven，

1.4 操作系统初始化配置

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
setenforce 0  # 临时

# 关闭swap
swapoff -a  # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

# 根据规划设置主机名
hostnamectl set-hostname <hostname>

# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.0.201 k8s-master
192.168.0.202 k8s-node1
192.168.0.203 k8s-node2
192.168.0.229 harbor.scajy.org
EOF

# 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # 生效

# 时间同步
yum install ntpdate -y
ntpdate time.windows.com

2. 部署Etcd集群

Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为解决Etcd单点故障，应采用集群方式部署，这里使用3台组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。

节点名称	IP
etcd-1	192.168.0.201
etcd-2	192.168.0.202
etcd-3	192.168.0.203

注：为了节省机器，这里与K8s节点机器复用。也可以独立于k8s集群之外部署，只要apiserver能连接到就行。

3. 安装docker【所有节点】

Kubernetes默认CRI（容器运行时）为Docker，因此先安装Docker。

3.1 安装docker

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install -y yum-utils device-mapper-persistent-data   lvm2
yum -y install docker-ce
systemctl enable docker && systemctl start docker

3.2 配置docker镜像下载加速器

cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

3.3 安装docker的tab补全功能

yum install -y bash-completion

3.4 重启docker

systemctl daemon-reload  && systemctl restart docker

4. 安装kubeadm/kubelet 【所有节点】

4.1 配置添加阿里云YUM软件源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

4.2 安装kubeadm，kubelet和kubectl

由于版本更新频繁，这里指定版本号安装

 yum install -y kubelet-1.19.0 kubeadm-1.19.0 kubectl-1.19.0
 systemctl enable kubelet

5. 部署kubernetes Master

参考部署文档

https://kubernetes.io/zh/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#initializing-your-control-plane-node

5.1 在master上执行

kubeadm init \
  --apiserver-advertise-address=192.168.0.201 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.19.0 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all

--apiserver-advertise-address 集群通告地址
--image-repository 由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址
--kubernetes-version K8s版本，与上面安装的一致
--service-cidr 集群内部虚拟网络，Pod统一访问入口
--pod-network-cidr Pod网络，，与下面部署的CNI网络组件yaml中保持一致

5.2 执行配置

[root@k8s-master ~]# kubeadm init \
>   --apiserver-advertise-address=192.168.0.201 \
>   --image-repository registry.aliyuncs.com/google_containers \
>   --kubernetes-version v1.19.0 \
>   --service-cidr=10.96.0.0/12 \
>   --pod-network-cidr=10.244.0.0/16 \
>   --ignore-preflight-errors=all
W1125 15:27:14.705112   21592 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.0
[preflight] Running pre-flight checks
	[WARNING NumCPU]: the number of available CPUs 1 is less than the required 2
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.201]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.201 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.201 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.504596 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: f71sjf.y17zsxdsnm7pft5l
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.201:6443 --token f71sjf.y17zsxdsnm7pft5l \
    --discovery-token-ca-cert-hash sha256:2a1dcf2140a80f526be1858aebb140037565c5a1fdf52e190fb4e3074d9c7290

5.3 拷贝kubectl使用的连接k8s认证文件到默认路径：

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

6. 加入kubernetes node节点

在所有node节点上执行，向集群添加新节点，执行在kubeadm init输出的kubeadm join命令：

kubeadm join 192.168.0.201:6443 --token f71sjf.y17zsxdsnm7pft5l \
    --discovery-token-ca-cert-hash sha256:2a1dcf2140a80f526be1858aebb140037565c5a1fdf52e190fb4e3074d9c7290

7. 部署容器网络（CNI）

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

注意：只需要部署下面其中一个，推荐Calico。

Calico是一个纯三层的数据中心网络方案，Calico支持广泛的平台，包括Kubernetes、OpenStack等。

Calico 在每一个计算节点利用 Linux Kernel 实现了一个高效的虚拟路由器（ vRouter）来负责数据转发，而每个 vRouter 通过 BGP 协议负责把自己上运行的 workload 的路由信息向整个 Calico 网络内传播。

此外，Calico 项目还实现了 Kubernetes 网络策略，提供ACL功能。

https://docs.projectcalico.org/getting-started/kubernetes/quickstart

7.1 下载容器网络配置

wget https://docs.projectcalico.org/manifests/calico.yaml

下载完后还需要修改里面定义

- name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"       # 这里的IP为前面的kubeadm初始化里面的pod IP

7.2 部署网络

[root@k8s-master ~]# kubectl apply -f calico.yaml 
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

7.3 查看部署

[root@k8s-master ~]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5dc87d545c-nscfb   1/1     Running   0          2m12s
calico-node-j6rhw                          0/1     Running   0          2m12s
calico-node-n7d6s                          1/1     Running   0          2m11s
calico-node-x86s2                          0/1     Running   0          2m11s
coredns-6d56c8448f-hkgnk                   1/1     Running   0          156m
coredns-6d56c8448f-jfbjs                   1/1     Running   0          156m
etcd-k8s-master                            1/1     Running   0          157m
kube-apiserver-k8s-master                  1/1     Running   0          157m
kube-controller-manager-k8s-master         1/1     Running   0          157m
kube-proxy-fhgbd                           1/1     Running   0          99m
kube-proxy-l7q4r                           1/1     Running   0          156m
kube-proxy-qwpjp                           1/1     Running   0          99m
kube-scheduler-k8s-master                  1/1     Running   0          157m

8. 部署dashboard

8.1 下载dashboard配置文件

下载配置文件

[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml

8.2 修改dashboard配置文件，暴露对外端口

修改默认dashboard只能集群内部访问，修改service为nodePort类型，暴露到外部：

vi recommended.yaml
...       #... 代表隐藏
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  selector:
    k8s-app: kubernetes-dashboard
  type: NodePort
...

8.3 启动dashboard服务

[root@k8s-master ~]# kubectl apply -f recommended.yaml 
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
The Service "kubernetes-dashboard" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

8.4 查看服务

[root@k8s-master ~]# kubectl get pods -n kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-7b59f7d4df-fbkg6   1/1     Running   0          85s
kubernetes-dashboard-5dbf55bd9d-spd9q        1/1     Running   0          87s

8.5 浏览器访问

访问地址：https://NodeIP:30001

8.6 配置用户角色

创建service account并绑定默认cluster-admin管理员集群角色：

# 创建用户
kubectl create serviceaccount dashboard-admin -n kube-system


# 用户授权
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin


# 获取用户Token
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

执行角色权限，

[root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created

[root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created


[root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:         dashboard-admin-token-ckqml
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 6c7b07d3-d3e3-4cd6-ad37-4b60de7c1a3d

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6InhnSERNcGRHaDV1TXBmay1SQ2NIenlLVHRlSi1uNTV0TnBBMHhHSFN6YmcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tY2txbWwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNmM3YjA3ZDMtZDNlMy00Y2Q2LWFkMzctNGI2MGRlN2MxYTNkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.czvP9MCh-uki7xWfrGfCt7IMNVYr1oNd2r10phUS8JFu4pMUmjGSq2M5mff_J-i5CmlC7tIzZRC1ZsA4qi9E00hedcl2zJ523Kiup_EcEUt-X-B5iCOgeNY38cc_f5u61_4y2jSmqayalTHy2ICX7_2aN8gQcC1r1iUf2Gy-Snm3Z9zsuZZxYD9VnUe0CLwiBCqYETn43vYvQx-1D6dKNqFWxNbDEcJWlLYbYTqKx0fNsTCucgXdH3pJzJ5Qx_3FgKPZO9Tjub-pkkFqsMD0S0WbFUxAwkyjBj1PK4DgoAwGcE2VS1wNu_bhbM7M4izIv6jz8wcn1cwWJfvAkfWGbg
ca.crt:     1066 bytes
namespace:  11 bytes

9. 集群基本验证

验证pod工作
验证pod网络通信
验证DNS解析

在kubernetes集群中创建一个pod，验证是否正常运行：

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc

浏览器访问测试

posted @ 2021-02-07 09:14 七月流星雨阅读(837) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

kubeadm快速部署一个k8s集群

kubeadm快速部署一个k8s集群

1. 前置知识点

1.1 生产环境可部署kubernetes集群的两种方式

1.2 安装要求

1.3 环境准备

1.3.1 软件环境

1.3.2 单master架构图

1.3.3 服务器整体规划

1.4 操作系统初始化配置

2. 部署Etcd集群

3. 安装docker【所有节点】

3.1 安装docker

3.2 配置docker镜像下载加速器

3.3 安装docker的tab补全功能

3.4 重启docker

4. 安装kubeadm/kubelet 【所有节点】

4.1 配置添加阿里云YUM软件源

4.2 安装kubeadm，kubelet和kubectl

5. 部署kubernetes Master

5.1 在master上执行

5.2 执行配置

5.3 拷贝kubectl使用的连接k8s认证文件到默认路径：

6. 加入kubernetes node节点

7. 部署容器网络（CNI）

7.1 下载容器网络配置

7.2 部署网络

7.3 查看部署

8. 部署dashboard

8.1 下载dashboard配置文件

8.2 修改dashboard配置文件，暴露对外端口

8.3 启动dashboard服务

8.4 查看服务

8.5 浏览器访问

8.6 配置用户角色

9. 集群基本验证

公告