第1章 K8S环境部署

Kubernetes部署方式

官方提供Kubernetes部署3种方式

• minikube

　　Minikube.是一个工具，可以在本地快速运行一个单点的Kubernetes，尝试Kubernetes或日常开发的用户使用。不能用于生产环境。

　　官方文档: https://kubernetes.io/docs/setup/minikube/

• 二进制包

　　从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。目前企业生产环境中主要使用该方式。

　　下载地址:    https:/github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113

• Kubeadm

　　Kubeadm是谷歌推出的一个专门用于快速部署kubernetes、集群的工具。在集群部署的过程中，可以通过kubeadm.init来初始化 master节点，然后使用kubeadm join将其他的节点加入到集群中。

　　Kubeadm通过简单配置可以快速将一个最小可用的集群运行起来。它在设计之初关注点是快速安装并将集群运行起来，而不是一步步关于各节点环境的准备工作。同样的，kubernetes集群在使用过程中的各种插件也不是kubeadm关注的重点，比如kubernetes集群 WEB Dashboard、prometheus监控集群业务等。kubeadm应用的日的是作为所有部署的基础，并通过kubeadm使得部署kubernetes集群更加容易。

 

　　Kubeadm的简单快捷的部署可以应用到如下三方面:

• l  新用户可以从kubeadm开始快速搭建Kubernete并了解。
• l  熟悉Kubernetes的用户可以使用kubeadm快速搭建集群并测试他们的应用。
• l  大型的项目可以将kubeadm配合其他的安装工具一起使用，形成一个比较复杂的系统。

　　官方文档:https://kubernetes.io/docs/setup/independent/install-kubeadm/

 

 原理：

                         

•  通过kubectl命令下发一些任务

  kubectl：命令管理工具

  • 下发任务需要通过验证（Auth）
  • 然后到达API服务器，Scheduler推算任务下发后的分配工作，Controler manager控制稳定的应用台数

                   Etcd为存储器，存储的是控制信息

                   Scheduler完成分配工作

                  Controler manager是控制操作（比如想跑3台nginx，down掉一台，这时控制器会在增加一台）

  • API server将任务下发给kubelet

                  被监控主机要安装kubelet,是接受API下发的任务,配合API工作

  • Kubele接受到各自的分任务之后将分发给podZ执行

                  一个pod中有一个容器或者是多个容器

  • 通过proxy代理将完后的任务通过防火墙发送到网络中

   

基于kubeadm 部署k8s集群

 

1、环境准备

主机ip	主机名	组件
192.168.2.111	k8s-master或nfs	kubeadm、kubelet、kubectl、docker-ce
192.168.2.112	k8s-node-1	kebeadm、kubelet、kebectl、docker-ce
192.168.2.113	k8s-node-2	kebeadm、kubelet、kubectl、docker-ce

　　　　　　　　所有主机配置推荐CPU 2C+   Memory 2G+

2.主机初始化配置

 

• 所有主机配置禁用防火墙和selinux

•  iptables -F
  
  systemctl stop firewalld
  
  systemctl disable firewalld
  
  setenforce 0
  
  sed -i '7c SELINUX=disabled' /etc/sysconfig/selinux
  
  systemctl stop NetworkManager
  

  　

 

• 配置主机名并绑定hosts

• [root@node-1 ~]# hostname k8s-master
  
  [root@node-1 ~]# bash
  
  [root@master ~]# cat << EOF >> /etc/hosts
  
  192.168.2.111 k8s-master
  
  192.168.2.112 k8s-node-1
  
  192.168.2.113 k8s-node-2
  
  EOF
  
  [root@master ~]# scp /etc/hosts 192.168.2.112:/etc/
  
  [root@master ~]# scp /etc/hosts 192.168.2.113:/etc/
  
   
  
  [root@node-1 ~]# hostname k8s-node-1
  
  [root@node-1 ~]# bash
  
   
  
  [root@node-2 ~]# hostname k8s-node-2
  
  [root@node-2 ~]# bash
  

• 安装基本软件包（三台都需要）

• [root@k8s-master ~]# yum install -y vim wget net-tools lrzsz
  

• 主配置初始化（三台都需要）

• [root@k8s-master ~]# swapoff -a
  [root@k8s-master ~]# sed -i '/swap/s/^/#/' /etc/fstab
  [root@k8s-master ~]# cat << EOF >> /etc/sysctl.conf
  > net.ipv4.ip_forward = 1
  > net.bridge.bridge-nf-call-ip6tables = 1
  > net.bridge.bridge-nf-call-iptables = 1
  > EOF
  　　如果sysctl -p 报错，需加载改模块
  
  　　[root@k8s-master ~]# modprobe br_netfilter
  

  　　

• 三台主机分别部署docker环境，因为kubernetes读容器的编排需要Docker的支持（安装过程忽略）

• 配置阿里云的源（所有主机都需要）

• [root@k8s-master ~]# cat << EOF > /etc/yum.repos.d/kubernetes.repo
   [kubernetes]
  name=Kubernetes
  baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
  enabled=1
  gpgcheck=1
  repo_gpgcheck=1
  gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
  https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  EOF
  [root@k8s-master ~]#  ls /etc/yum.repos.d/
  bak               docker-ce.repo  kubernetes.repo
  CentOS-Base.repo  epel.repo       local.repo
  

• 安装三个工具包（所有主机都需要）且设置为开机自启动

• [root@k8s-master ~]#  yum -y install kubectl-1.17.0 kubeadm-1.17.0 kubelet-1.17.0

  　　kubelet刚安装完成后，迪过systemctl start kubelet,方式是无法启动的，需要加入节点或初始化为master后才可启动成功。

         如果在命令执行过程中出现索引gpg检查失败的情况，请使用yum -y install --nogpgcheck kubelet kubeadm kubectl来安装。

  
  [root@k8s-master ~]# yum -y install --nogpgcheck kubectl-1.17.0 kubeadm-1.17.0 kubelet-1.17.0
  [root@k8s-master ~]# systemctl enable kubelet
           Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

• 在master端配置init-config.yaml文件
• Kubeadm提供了很多配置项,Kubeadm配置在Kubernetes集群中是存储在ConfigMapg,中的，也可将这些配置写入配置文件，方便管理复杂的配置项。Kubeadm配内容是通过kubeadm config命令写入配置文件的。
• 上传initt-config.yaml文件

• init-config.yaml链接：https://pan.baidu.com/s/1BUuE4LkyYcxI_fyaEov8gg         提取码：m86t 

  编辑文件：

  advertiseAddress:为自己的master的IP地址

  kubernetesVersion: v1.17.0

• [root@k8s-master ~]# vim init-config.yaml 
  [root@k8s-master ~]# cat init-config.yaml 
  apiVersion: kubeadm.k8s.io/v1beta2
  bootstrapTokens:
  - groups:
    - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
    - signing
    - authentication
  kind: InitConfiguration
  localAPIEndpoint:
    advertiseAddress: 192.168.2.111
    bindPort: 6443
  nodeRegistration:
    criSocket: /var/run/dockershim.sock
    name: k8s-master
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
  ---
  apiServer:
    timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta2
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  controllerManager: {}
  dns:
    type: CoreDNS
  etcd:
    local:
      dataDir: /var/lib/etcd
  imageRepository: registry.aliyuncs.com/google_containers
  kind: ClusterConfiguration

  kubernetesVersion: v1.17.0

  networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 scheduler: {}

• 安装master节点

• [root@k8s-master ~]# kubeadm config images pull --config init-config.yaml
  W1222 15:13:28.961040 17823 validation.go:28] Cannot validate kube-proxy config - no validator is available
  W1222 15:13:28.961091 17823 validation.go:28] Cannot validate kubelet config - no validator is available
  [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.17.0
  [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.17.0
  [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.17.0
  [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.17.0
  [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.1
  [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.4.3-0
  [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.6.5
  [root@k8s-master ~]#

• 查看镜像

• [root@k8s-master ~]# docker images
  REPOSITORY                                                        TAG       IMAGE ID       CREATED         SIZE
  registry.aliyuncs.com/google_containers/kube-proxy                v1.20.0   10cc881966cf   13 days ago     118MB
  registry.aliyuncs.com/google_containers/kube-apiserver            v1.20.0   ca9843d3b545   13 days ago     122MB
  registry.aliyuncs.com/google_containers/kube-scheduler            v1.20.0   3138b6e3d471   13 days ago     46.4MB
  registry.aliyuncs.com/google_containers/kube-controller-manager   v1.20.0   b9fa1895dcaa   13 days ago     116MB
  centos                                                            7         8652b9f0cb4c   5 weeks ago     204MB
  registry.aliyuncs.com/google_containers/coredns                   1.6.5     70f311871ae1   13 months ago   41.6MB
  registry.aliyuncs.com/google_containers/etcd                      3.4.3-0   303ce5db0e90   14 months ago   288MB
  google/cadvisor                                                   latest    eb1210707573   2 years ago     69.6MB
  registry.aliyuncs.com/google_containers/pause                     3.1       da86e6ba6ca1   3 years ago     742kB
  tutum/influxdb                                                    latest    c061e5808198   4 years ago     290MB
  

• 初始化

• [root@k8s-master ~]# kubeadm init --config=init-config.yaml 

• 报错：

• 

   

• 解决方法：

• [root@k8s-master ~]# echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables
  [root@k8s-master ~]# kubeadm init --config=init-config.yaml 
  

• 初始化成功

• 

• 显示成功后会出现三个命令，直接将三条命令执行一下，初始化就成功了

• [root@k8s-master ~]#  mkdir -p $HOME/.kube
  [root@k8s-master ~]#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  [root@k8s-master ~]#   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  [root@k8s-master ~]# 
  

• 初始化过程中做了什么？
  

  kubeadm init主要执行了以下操作:

  [init]:指定版本进行初始化操作

  [preflight] :初始化前的检查和下载所需要的Docker镜像文件

  [kubelet-start]:生成 kubelet的配置文件”./var/lib/kubelet/config.xam，没有这个文件

  kubelet无法启动，所以初始化之前的 kubelet实际上启动失败。

  [certificates]:生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。

  [kubecanfig]:生成 Kubeconfig文件，存放在/et/kubernetes目录中，组件之间通信需要使用对应文件。

  [control-plane]:使用/etc/kubernetes/manifest日录下的YAML_文件，安装Master组件。

  [etcd]:使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。

  [wait-control-plane]:等待control-plan部署的Master组件启动。

  [apiclient]:检查Master组件服务状态。

  [uploadconfig]:更新配置

  [kubelet]:使用configMap配置 kubelet。

  [patchnode]:更新CNI信息到Node 上，通过注释的方式记录。

  [mark-control-plane]:为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。

  [bootstrap-token]:生成token记录下来，后边使用kubeadm join往集群中添加节点时

3.添加节点

• 查看节点

• [root@k8s-master ~]# kubectl get nodes
  NAME         STATUS     ROLES    AGE   VERSION
  k8s-master   NotReady   master   29m   v1.17.0
  [root@k8s-master ~]# 
  

• 添加nod节点（将初始化中的命令复制到节点服务器上，两台）

• [root@k8s-node-1 ~]# kubeadm join 192.168.2.111:6443 --token abcdef.0123456789abcdef     --discovery-token-ca-cert-hash sha256:15efa45232e702325683e8f734bc2a50559311cd8521087687dbddfbbccf0fb7 
  

• 再次查看master（将会出现两个节点）

• [root@k8s-master ~]# kubectl get nodes
  NAME         STATUS     ROLES    AGE   VERSION
  k8s-master   NotReady   master   38m   v1.17.0
  k8s-node-1   NotReady   <none>   86s   v1.17.0
  k8s-node-2   NotReady   <none>   3s    v1.17.0
  

 

4.安装flannel

 

　　Master节点NotReady的原因就是因为没有使用任何的网络插件,此时Node和Master的连接还不正常。目前最流行的Kubernetes网络插件有Flannel、Calico、Canal、Weave这里选择使用flannel。

　　安装包链接：https://pan.baidu.com/s/1qsZnAkCK7F08iltJVsKEgg
　　提取码：5lmo

• master上传kube-flannet.yml
• 所有主机上传flannel_v0.12.0-amd64.tar
• 所有节点主机的操作

• [root@k8s-node-1 ~]# docker load < flannel_v0.12.0-amd64.tar 
  256a7af3acb1: Loading layer [==================================================>]  5.844MB/5.844MB
  d572e5d9d39b: Loading layer [==================================================>]  10.37MB/10.37MB
  57c10be5852f: Loading layer [==================================================>]  2.249MB/2.249MB
  7412f8eefb77: Loading layer [==================================================>]  35.26MB/35.26MB
  05116c9ff7bf: Loading layer [==================================================>]   5.12kB/5.12kB
  Loaded image: quay.io/coreos/flannel:v0.12.0-amd64
  

  　　

• master的操作

• [root@k8s-master ~]# kubectl apply -f kube-flannel.yml 
  podsecuritypolicy.policy/psp.flannel.unprivileged created
  clusterrole.rbac.authorization.k8s.io/flannel created
  clusterrolebinding.rbac.authorization.k8s.io/flannel created
  serviceaccount/flannel created
  configmap/kube-flannel-cfg created
  daemonset.apps/kube-flannel-ds-amd64 created
  daemonset.apps/kube-flannel-ds-arm64 created
  daemonset.apps/kube-flannel-ds-arm created
  daemonset.apps/kube-flannel-ds-ppc64le created
  daemonset.apps/kube-flannel-ds-s390x created
  

  　　

• 查看是否搭建成功

• [root@k8s-master ~]# kubectl get nodes
  NAME         STATUS   ROLES    AGE   VERSION
  k8s-master   Ready    master   72m   v1.17.0
  k8s-node-1   Ready    <none>   34m   v1.17.0
  k8s-node-2   Ready    <none>   33m   v1.17.0
  

  　　

• [root@k8s-master ~]# kubectl get pods -n kube-system
  NAME                                 READY   STATUS    RESTARTS   AGE
  coredns-9d85f5447-d9vgg              1/1     Running   0          68m
  coredns-9d85f5447-spptr              1/1     Running   0          68m
  etcd-k8s-master                      1/1     Running   0          69m
  kube-apiserver-k8s-master            1/1     Running   0          69m
  kube-controller-manager-k8s-master   1/1     Running   0          69m
  kube-flannel-ds-amd64-cgxk4          1/1     Running   0          21m
  kube-flannel-ds-amd64-cjd9z          1/1     Running   0          21m
  kube-flannel-ds-amd64-vfb6r          1/1     Running   0          21m
  kube-proxy-czpcx                     1/1     Running   0          32m
  kube-proxy-vn5jp                     1/1     Running   0          68m
  kube-proxy-w7g65                     1/1     Running   0          30m
  kube-scheduler-k8s-master            1/1     Running   0          69m
  [root@k8s-master ~]# kubectl get pods -A -n kube-system
  NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
  kube-system   coredns-9d85f5447-d9vgg              1/1     Running   0          69m
  kube-system   coredns-9d85f5447-spptr              1/1     Running   0          69m
  kube-system   etcd-k8s-master                      1/1     Running   0          69m
  kube-system   kube-apiserver-k8s-master            1/1     Running   0          69m
  kube-system   kube-controller-manager-k8s-master   1/1     Running   0          69m
  kube-system   kube-flannel-ds-amd64-cgxk4          1/1     Running   0          21m
  kube-system   kube-flannel-ds-amd64-cjd9z          1/1     Running   0          21m
  kube-system   kube-flannel-ds-amd64-vfb6r          1/1     Running   0          21m
  kube-system   kube-proxy-czpcx                     1/1     Running   0          32m
  kube-system   kube-proxy-vn5jp                     1/1     Running   0          69m
  kube-system   kube-proxy-w7g65                     1/1     Running   0          30m
  kube-system   kube-scheduler-k8s-master            1/1     Running   0          69m
  

   

• 至此k8s集群环境搭建成功

 基于二进制部署K8S

　　相关安装包

• 链接：https://pan.baidu.com/s/1Jmk4L1H2bLkFm-ryx7xzew
• 提取码：m3dg

 1.docker安装完毕之后进行如下的配置

• 还是上面的三台虚拟机将三台虚拟机的名字分别改为以下三个

• hostname k8s-master
  bash
  
  
   hostname k8s-node01
  bash
  
  
  hostname k8s-node02
   bash
  

• 编写host文件，三台都需要，以master为例子,内容如下

• [root@k8s-master ~]#  cat << EOF >> /etc/hosts
  192.168.2.111 k8s-master
  192.168.2.112 k8s-node01
  192.168.2.113 k8s-node02
  EOF
  

2.证书

　　k8s系统各个组件之间需要使用TLS证书进行通信，下列我们将使用CloudFlare的PKI工具集CFSSL来生成Certificate Authority 和其他证书。

• master主机上安装证书生成工具

• mkdir -p /root/software/ssl
  cd /root/software/ssl/
  [root@k8s-master ssl]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
  [root@k8s-master ssl]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
  [root@k8s-master ssl]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
  [root@k8s-master ssl]# chmod +x * //下载完后设置执行权限
  [root@k8s-master ssl]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
  [root@k8s-master ssl]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
  [root@k8s-master ssl]# mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
  

• 查看是否成功

•  cfssl --help
  Usage:
  Available commands:
  	sign
  	version
  	ocspdump
  	ocspserve
  	scan
  	bundle
  	genkey
  	gencrl
  	ocsprefresh
  	selfsign
  	info
  	serve
  	revoke
  	certinfo
  	gencert
  	ocspsign
  	print-defaults
  Top-level flags:
    -allow_verification_with_non_compliant_keys
      	Allow a SignatureVerifier to use keys which are technically non-compliant with RFC6962.
    -loglevel int
      	Log level (0 = DEBUG, 5 = FATAL) (default 1)
  

• master主机编写证书相关的json文件（脚本）用来生成证书（一共4个文件，生成八个证书）

• [root@k8s-master ~]# cat << EOF > ca-config.json
  {
    "signing": {
      "default": {
        "expiry": "87600h"
      },
      "profiles": {
        "kubernetes": {
          "expiry": "87600h",
          "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
          ]
        }
      }
    }
  }
  EOF
  
  [root@k8s-master ~]# cat << EOF > ca-csr.json
  {
    "CN": "kubernetes",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "L": "Beijing",
        "ST": "Beijing",
        "O": "k8s",
        "OU": "System"
      }
    ]
  }
  EOF
  [root@k8s-master ~]# cat << EOF > server-csr.json
  {
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.2.111",
      "192.168.2.112",
      "192.168.2.113",
      "10.10.10.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "L": "BeiJing",
        "ST": "BeiJing",
        "O": "k8s",
        "OU": "System"
      }
    ]
  }
  EOF
  [root@k8s-master ~]# cat << EOF > admin-csr.json
  {
    "CN": "admin",
    "hosts": [],
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "L": "BeiJing",
        "ST": "BeiJing",
        "O": "system:masters",
        "OU": "System"
      }
    ]
  }
  EOF
  

• 将四个文件进行 pem 证书的生成

• [root@k8s-master ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
  
  [root@k8s-master ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
  
  [root@k8s-master ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
  
  [root@k8s-master ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

• 删除证书以外的 json 文件，只保留 pem 证书　

• [root@k8s-master ssl]# ls | grep -v pem | xargs -i rm {}
  //删除证书以外的 json 文件，只保留 pem 证书
  
  [root@k8s-master ssl]# ls -l
  总用量 32
  -rw------- 1 root root 1675 11月 13 23:09 admin-key.pem
  -rw-r--r-- 1 root root 1399 11月 13 23:09 admin.pem
  -rw------- 1 root root 1679 11月 13 23:03 ca-key.pem
  -rw-r--r-- 1 root root 1359 11月 13 23:03 ca.pem
  -rw------- 1 root root 1675 11月 13 23:12 kube-proxy-key.pem
  -rw-r--r-- 1 root root 1403 11月 13 23:12 kube-proxy.pem
  -rw------- 1 root root 1679 11月 13 23:07 server-key.pem
  -rw-r--r-- 1 root root 1627 11月 13 23:07 server.pem
  

3.创建k8s目录，部署etcd

• 上传etcd包，解压包，并拷贝二进制bin文件将命令进行部署

• mkdir /opt/kubernetes
  mkdir /opt/kubernetes/{bin,cfg,ssl}
  tar xf etcd-v3.3.18-linux-amd64.tar.gz 
  cd etcd-v3.3.18-linux-amd64/
  mv etcd /opt/kubernetes/bin/
  mv etcdctl /opt/kubernetes/bin/
  

• 在master主机创建/opt/kubernetes/cfg/etcd文件，这个文件是etcd的配置文件

  [root@k8s-master etcd-v3.3.18-linux-amd64]# cat /opt/kubernetes/cfg/etcd
  #[Member]
  ETCD_NAME="etcd01"
  ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
  ETCD_LISTEN_PEER_URLS="https://192.168.2.111:2380"
  ETCD_LISTEN_CLIENT_URLS="https://192.168.2.111:2379"
  #[Clustering]
  ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.2.111:2380"
  ETCD_ADVERTISE_CLIENT_URLS="https://192.168.2.111:2379"
  ETCD_INITIAL_CLUSTER="etcd01=https://192.168.2.111:2380,etcd02=https://192.168.2.112:2380,etcd03=https://192.168.2.113:2380"
  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  ETCD_INITIAL_CLUSTER_STATE="new"
  

• 创建启动脚本

• [root@k8s-master etcd-v3.3.18-linux-amd64]# cat /usr/lib/systemd/system/etcd.service
  [Unit]
  Description=Etcd Server
  After=network.target
  After=network-online.target
  Wants=network-online.target
  [Service]
  Type=notify
  EnvironmentFile=-/opt/kubernetes/cfg/etcd
  ExecStart=/opt/kubernetes/bin/etcd \
  --name=${ETCD_NAME} \
  --data-dir=${ETCD_DATA_DIR} \
  --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --initial-cluster=${ETCD_INITIAL_CLUSTER} \
  --initial-cluster-token=${ETCD_INITIAL_CLUSTER} \
  --initial-cluster-state=new \
  --cert-file=/opt/kubernetes/ssl/server.pem \
  --key-file=/opt/kubernetes/ssl/server-key.pem \
  --peer-cert-file=/opt/kubernetes/ssl/server.pem \
  --peer-key-file=/opt/kubernetes/ssl/server-key.pem \
  --trusted-ca-file=/opt/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/opt/kubernetes/ssl/ca.pem
  Restart=on-failure
  LimitNOFILE=65536
  [Install]
  WantedBy=multi-user.target
  

  　　

• 在master主机拷贝一份etcd脚本所依赖的证书

• [root@k8s-master etcd-v3.3.18-linux-amd64]# cd /root/software/
  [root@k8s-master software]# cp ssl/server*pem ssl/ca*.pem /opt/kubernetes/ssl/
  

• 重新启动etcd(这个时候会卡死，但是无所谓，直接ctrl+c退出，卡的原因开始节点未连接)

• [root@k8s-master software]# systemctl start etcd
  [root@k8s-master software]# systemctl enable etcd
  

• 查看进程，只要进程在就可以

• [root@k8s-master software]# ps aux | grep etcd
  

• 在node1和node2也需要配置etcd文件，修改完的配置如下

• 在node-01主机创建/opt/kubernetes/cfg/etcd文件，并写如下内容
  
  #[Member]
  ETCD_NAME="etcd02"
  ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
  ETCD_LISTEN_PEER_URLS="https://192.168.2.112:2380"
  ETCD_LISTEN_CLIENT_URLS="https://192.168.2.112:2379"
  #[Clustering]
  ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.2.112:2380"
  ETCD_ADVERTISE_CLIENT_URLS="https://192.168.2.112:2379"
  ETCD_INITIAL_CLUSTER="etcd01=https://192.168.2.111:2380,etcd02=https://192.168.2.112:2380,etcd03=https://192.168.2.113:2380"
  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  ETCD_INITIAL_CLUSTER_STATE="new"
  
  在node-02主机创建/opt/kubernetes/cfg/etcd文件，并写如下内容
  
  
  #[Member]
  ETCD_NAME="etcd03"
  ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
  ETCD_LISTEN_PEER_URLS="https://192.168.2.113:2380"
  ETCD_LISTEN_CLIENT_URLS="https://192.168.2.113:2379"
  #[Clustering]
  ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.2.113:2380"
  ETCD_ADVERTISE_CLIENT_URLS="https://192.168.2.113:2379"
  ETCD_INITIAL_CLUSTER="etcd01=https://192.168.2.111:2380,etcd02=https://192.168.2.112:2380,etcd03=https://192.168.2.113:2380"
  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  ETCD_INITIAL_CLUSTER_STATE="new"
  

  　　

• master将启动脚本也传给节点主机

• [root@k8s-master ~]# scp /usr/lib/systemd/system/etcd.service 192.168.2.112:/usr/lib/systemd/system/    
  [root@k8s-master ~]# scp /usr/lib/systemd/system/etcd.service 192.168.2.113:/usr/lib/systemd/system/
  

  　　

• 节点主机分别重新启动etcd

• [root@k8s-node01 ~]# systemctl start etcd
  systemctl enable etcd
  Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
  [root@k8s-node02 ~]# systemctl start etcd
  [root@k8s-node02 ~]# systemctl enable etcd
  Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
  

• 将etcd的命令添加在全局的环境变量中

• [root@k8s-master ~]# vim /etc/profile
  export PATH=$PATH:/opt/kubernetes/bin
  [root@k8s-master ~]# source /etc/profile
  

• 查看etcd集群的部署，会有三个节点

• [root@k8s-master ~]# cd /root/software/ssl/
  
  [root@k8s-master ssl]# etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.2.111:2379,https://192.168.2.112,https://192.168.2.113:2379" cluster-health                          
  

　　至此etcd成功部署。

4.部署Flannel网络

　　flannel是overlay网络中的一种，也是将原数据包封装在另一种网络包里进行路由转换和通信。

• 在主节点写入分配子网段到etcd，供flanneld使用

• [root@k8s-master ~]# cd /root/software/ssl/
  [root@k8s-master ssl]# etcdctl -ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.2.111:2379,https://192.168.2.112:2379,https://192.168.2.113:2379" set /coreos.com/network/config '{"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"} }'
  输出结果：
  {"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"} }
  

• 上传flannel包，解压包并拷贝到node节点

• [root@k8s-master ~]# tar xf flannel-v0.12.0-linux-amd64.tar.gz 
  [root@k8s-master ~]# scp flanneld mk-docker-opts.sh 192.168.2.112:/opt/kubernetes/bin/
  [root@k8s-master ~]# scp flanneld mk-docker-opts.sh 192.168.2.113:/opt/kubernetes/bin/
  

• 在 k8s-node1 与 k8s-node2 主机上分别编辑 flanneld 配置文件。下面以 k8s-node1 为例进行操作演示。

• [root@k8s-node01 ~]# vim /opt/kubernetes/cfg/flanneld
  FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.2.111:2379,https://192.168.2.112:2379,https://192.168.2.113:2379 -etcd-cafile=/opt/kubernetes/ssl/ca.pem -etcd-certfile=/opt/kubernetes/ssl/server.pem -etcd-keyfile=/opt/kubernetes/ssl/server-key.pem"
  
  [root@k8s-node1 ~]# cat <<EOF >/usr/lib/systemd/system/flanneld.service
  [Unit]
  Description=Flanneld overlay address etcd agent
  After=network-online.target network.target
  Before=docker.service
  [Service]
  Type=notify
  EnvironmentFile=/opt/kubernetes/cfg/flanneld
  ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
  ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
  Restart=on-failure
  [Install]
  WantedBy=multi-user.target
  EOF
  

• 在 k8s-node1 与 k8s-node2 主机上配置 Docker 启动指定网段，修改 Docker 配置脚本文件。下面以 k8s-node1 为例进行操作演示。（将原有ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock）进行注释，添加下面的两行。

• [root@k8s-node01 ~]# vim /usr/lib/systemd/system/docker.service
  EnvironmentFile=/run/flannel/subnet.env
  ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
  

• 两台节点都需要重新启动flanneld

• systemctl start flanneld
  systemctl enable flanneld
  systemctl daemon-reload
  systemctl restart docker
  

• 查看一下相应的网络,docker和flannel在同一网段

• [root@k8s-node01 ~]# ifconfig
  docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
          inet 172.17.96.1  netmask 255.255.255.0  broadcast 172.17.96.255
          ether 02:42:53:3a:56:7f  txqueuelen 0  (Ethernet)
          RX packets 0  bytes 0 (0.0 B)
          RX errors 0  dropped 0  overruns 0  frame 0
          TX packets 0  bytes 0 (0.0 B)
          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
  
  flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
          inet 172.17.96.0  netmask 255.255.255.255  broadcast 0.0.0.0
          inet6 fe80::1430:4bff:fe88:a4a9  prefixlen 64  scopeid 0x20<link>
          ether 16:30:4b:88:a4:a9  txqueuelen 0  (Ethernet)
          RX packets 0  bytes 0 (0.0 B)
          RX errors 0  dropped 0  overruns 0  frame 0
          TX packets 0  bytes 0 (0.0 B)
          TX errors 0  dropped 68 overruns 0  carrier 0  collisions 0
  

• 在 k8s-node2 上测试到 node1 节点 docker0 网桥 IP 地址的连通性，出现如下结果说明Flanneld 安装成功。　

• [root@k8s-node02 ~]# ping 172.17.96.1
  PING 172.17.96.1 (172.17.96.1) 56(84) bytes of data.
  64 bytes from 172.17.96.1: icmp_seq=1 ttl=64 time=0.543 ms
  

  　至此 Node 节点的 Flannel 配置完成。

5.部署k8s-master组件

• 上传kubernetes-server-linux-amd64.tar.gz ，解压并添加在kubectl命令环境。

• [root@k8s-master ~]# tar xf kubernetes-server-linux-amd64.tar.gz 
  [root@k8s-master ~]# cd kubernetes/server/bin/
  [root@k8s-master bin]# cp kubectl /opt/kubernetes/bin/

• 创建TLS Booystrapping Token

• [root@k8s-master bin]# cd /opt/kubernetes/
  
  [root@k8s-master kubernetes]# export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
  [root@k8s-master kubernetes]# cat <<EOF > token.csv 
  ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
  EOF
  

• 创建kubelet kubeconfig

• [root@k8s-master kubernetes]# export KUBE_APISERVER="https://192.168.2.111:6443"
  
  设置集群参数
  [root@k8s-master kubernetes]# cd /root/software/ssl/
  [root@k8s-master ssl]# kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
  
  
  设置客户端认证参数
  [root@k8s-master ssl]# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig
  
  
  设置上下文参数
  [root@k8s-master ssl]# kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig
  
  设置默认上下文
  
  [root@k8s-master ssl]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
  

 

 

• 创建 kuby-proxy kubeconfig

 

• [root@k8s-master ssl]# kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
  
  [root@k8s-master ssl]# kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
  
  [root@k8s-master ssl]# kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
  
  [root@k8s-master ssl]# kubectl config use-context default \
  --kubeconfig=kube-proxy.kubeconfig
  

6.部署 Kube-apiserver

• [root@k8s-master ssl]# cd /root/kubernetes/server/bin/
  [root@k8s-master bin]# cp kube-controller-manager kube-scheduler kube-apiserver /opt/kubernetes/bin/
  [root@k8s-master bin]# cp /opt/kubernetes/token.csv /opt/kubernetes/cfg/
  [root@k8s-master bin]# cd /opt/kubernetes/bin
  

• 上传master.zip到当前目录

• [root@k8s-master bin]# unzip master.zip 
  Archive: master.zip
  inflating: scheduler.sh 
  inflating: apiserver.sh 
  inflating: controller-manager.sh
  
  [root@k8s-master bin]# chmod +x *.sh
  [root@k8s-master bin]# ./apiserver.sh 192.168.2.111 https://192.168.2.111:2379,https://192.168.2.112:2379,https://192.168.2.113:2379
  Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
  

 

7.部署kube-controller-manager

 

• [root@k8s-master bin]# sh controller-manager.sh 127.0.0.1
  Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
  

 

 

8.部署 kube-scheduler

• [root@k8s-master bin]# sh scheduler.sh 127.0.0.1
  Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
  

 

9.测试组件是否运行正常

• [root@k8s-master bin]# kubectl get cs
  NAME                 STATUS    MESSAGE             ERROR
  scheduler            Healthy   ok                  
  etcd-0               Healthy   {"health":"true"}   
  etcd-1               Healthy   {"health":"true"}   
  etcd-2               Healthy   {"health":"true"}   
  controller-manager   Healthy   ok 
  

 

10.部署K8s-node 组件

• 准备环境

• [root@k8s-master ~]# cd /root/software/ssl/
  [root@k8s-master ssl]# scp *kubeconfig 192.168.2.112:/opt/kubernetes/cfg/ 
  [root@k8s-master ssl]# scp *kubeconfig 192.168.2.113:/opt/kubernetes/cfg/
  
  [root@k8s-master ssl]# cd /root/kubernetes/server/bin
  [root@k8s-master bin]# scp kubelet kube-proxy 192.168.2.112:/opt/kubernetes/bin
  [root@k8s-master bin]# scp kubelet kube-proxy 192.168.2.113:/opt/kubernetes/bin
  
  [root@k8s-master bin]# kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap
  
  clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
  
  [root@k8s-master bin]# kubectl describe clusterrolebinding kubelet-bootstrap
  Name: kubelet-bootstrap
  Labels: <none>
  Annotations: <none>
  Role:
  Kind: ClusterRole
  Name: system:node-bootstrapper
  Subjects:
  Kind Name Namespace
  ---- ---- ---------
  User kubelet-bootstrap
  
  [root@k8s-node01 ~]# cd /opt/kubernetes/bin/
  上传node.zip
  [root@k8s-node01 bin]# unzip node.zip 
  Archive: node.zip
  inflating: kubelet.sh 
  inflating: proxy.sh 
  [root@k8s-node01 bin]# chmod +x *.sh
  [root@k8s-node01 bin]# sh kubelet.sh 192.168.2.112 192.168.2.254
  Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
  

• 执行以下命令，两个节点主机都需要

• [root@k8s-node02 ~]# cd /opt/kubernetes/bin/
  上传node.zip
  [root@k8s-node02 bin]# unzip node.zip 
  Archive: node.zip
  inflating: kubelet.sh 
  inflating: proxy.sh 
  [root@k8s-node02 bin]# chmod +x *.sh
  [root@k8s-node02 bin]# sh kubelet.sh 192.168.2.113 192.168.2.254
  Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
  

 

11.部署 kube-proxy

• 在两台node主机都需要执行

• [root@k8s-node01 bin]# sh proxy.sh 192.168.2.112
  Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
  
  
  [root@k8s-node02 bin]# sh proxy.sh 192.168.2.113
  Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
  

• 查看node节点组件是否安装成功

• [root@k8s-node01 bin]# ps -ef | grep kube
  [root@k8s-node02 bin]# ps -ef | grep kube
  

• 查看自动签发证书

• [root@k8s-master bin]# kubectl get csr
  NAME AGE REQUESTOR CONDITION
  node-csr-7M_L1gX2uGXM3prE3ruXM3IJsafgqYlpOI07jBpSjnI 3m2s kubelet-bootstrap Pending
  node-csr-FK7fRGabCBuX0W-Gt_ofM4VK5F_ZgNaIFsdEn1eVOq0 108s kubelet-bootstrap   Pending
  [root@k8s-master bin]# kubectl certificate approve node-csr-7M_L1gX2uGXM3prE3ruXM3IJsafgqYlpOI07jBpSjnI
  certificatesigningrequest.certificates.k8s.io/node-csr-7M_L1gX2uGXM3prE3ruXM3IJsafgqYlpOI07jBpSjnI approved
  [root@k8s-master bin]# kubectl certificate approve node-csr-FK7fRGabCBuX0W-Gt_ofM4VK5F_ZgNaIFsdEn1eVOq0
  certificatesigningrequest.certificates.k8s.io/node-csr-FK7fRGabCBuX0W-Gt_ofM4VK5F_ZgNaIFsdEn1eVOq0 approved
  

• 查看节点

• [root@k8s-master bin]# kubectl get node
  NAME STATUS ROLES AGE VERSION
  192.168.2.112 Ready <none> 50s v1.17.3
  192.168.2.113 Ready <none> 23s v1.17.3
  

   

　　至此k8s集群部署成功