搭建k8s高可用集群

搭建k8s高可用集群

一、kubernetes 概述

1、kubernetes 基本介绍

kubernetes，简称 K8s，是用 8 代替 8 个字符“ubernete”而成的缩写。是一个开源

的，用于管理云平台中多个主机上的容器化的应用，Kubernetes 的目标是让部署容器化的

应用简单并且高效（powerful）,Kubernetes 提供了应用部署，规划，更新，维护的一种

机制。

传统的应用部署方式是通过插件或脚本来安装应用。这样做的缺点是应用的运行、配

置、管理、所有生存周期将与当前操作系统绑定，这样做并不利于应用的升级更新/回滚等

操作，当然也可以通过创建虚拟机的方式来实现某些功能，但是虚拟机非常重，并不利于

可移植性。

新的方式是通过部署容器方式实现，每个容器之间互相隔离，每个容器有自己的文件

系统 ，容器之间进程不会相互影响，能区分计算资源。相对于虚拟机，容器能快速部署，

由于容器与底层设施、机器文件系统解耦的，所以它能在不同云、不同版本操作系统间进

行迁移。

容器占用资源少、部署快，每个应用可以被打包成一个容器镜像，每个应用与容器间

成一对一关系也使容器有更大优势，使用容器可以在 build 或 release 的阶段，为应用创

建容器镜像，因为每个应用不需要与其余的应用堆栈组合，也不依赖于生产环境基础结构，

这使得从研发到测试、生产能提供一致环境。类似地，容器比虚拟机轻量、更“透明”，

这更便于监控和管理。

Kubernetes 是 Google 开源的一个容器编排引擎，它支持自动化部署、大规模可伸缩、

应用容器化管理。在生产环境中部署一个应用程序时，通常要部署该应用的多个实例以便

对应用请求进行负载均衡。

在 Kubernetes 中，我们可以创建多个容器，每个容器里面运行一个应用实例，然后通

过内置的负载均衡策略，实现对这一组应用实例的管理、发现、访问，而这些细节都不需

要运维人员去进行复杂的手工配置和处理。

2、**kubernetes 功能和架构**

2.1 概述

Kubernetes 是一个轻便的和可扩展的开源平台，用于管理容器化应用和服务。通过

Kubernetes 能够进行应用的自动化部署和扩缩容。在 Kubernetes 中，会将组成应用的容

器组合成一个逻辑单元以更易管理和发现。Kubernetes 积累了作为 Google 生产环境运行

工作负载 15 年的经验，并吸收了来自于社区的最佳想法和实践。

2.2 K8s 功能:

（1）自动装箱

基于容器对应用运行环境的资源配置要求自动部署应用容器

（2）自我修复(自愈能力)

当容器失败时，会对容器进行重启

当所部署的 Node 节点有问题时，会对容器进行重新部署和重新调度当容器未通过监控检查时，会关闭此容器直到容器正常运行时，才会对外提供服务

（3）水平扩展

通过简单的命令、用户 UI 界面或基于 CPU 等资源使用情况，对应用容器进行规模扩大

或规模剪裁

（3）服务发现

用户不需使用额外的服务发现机制，就能够基于 Kubernetes 自身能力实现服务发现和

负载均衡

（4）滚动更新

可以根据应用的变化，对应用容器运行的应用，进行一次性或批量式更新

（5）版本回退

可以根据应用部署情况，对应用容器运行的应用，进行历史版本即时回退

（6）密钥和配置管理

在不需要重新构建镜像的情况下，可以部署和更新密钥和应用配置，类似热部署。

（7）存储编排

自动实现存储系统挂载及应用，特别对有状态应用实现数据持久化非常重要

存储系统可以来自于本地目录、网络存储(NFS、Gluster、Ceph 等)、公共云存储服务

（8）批处理

提供一次性任务，定时任务；满足批量数据处理和分析的场景

2.3 应用部署架构分类

(1) 无中心节点架构

GlusterFS

(2) 有中心节点架构

HDFS

K8S

2.4 k8s 集群架构

 

2.5 k8s 集群架构节点角色功能

Master Node

k8s 集群控制节点，对集群进行调度管理，接受集群外用户去集群操作请求；

Master Node 由 API Server、Scheduler、ClusterState Store（ETCD 数据库）和

Controller MangerServer 所组成

Worker Node

集群工作节点，运行用户业务应用容器；

Worker Node 包含 kubelet、kube proxy 和 ContainerRuntime；

 

二、kubernetes 集群搭建(kubeadm 方式)

1、前置知识点

目前生产部署 Kubernetes 集群主要有两种方式：

（1）kubeadm

Kubeadm 是一个 K8s 部署工具，提供 kubeadm init 和 kubeadm join，用于快速部

署 Kubernetes 集群。

官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

（2）二进制包

从 github 下载发行版的二进制包，手动部署每个组件，组成 Kubernetes 集群。

Kubeadm 降低部署门槛，但屏蔽了很多细节，遇到问题很难排查。如果想更容易可

控，推荐使用二进制包部署 Kubernetes 集群，虽然手动部署麻烦点，期间可以学习很

多工作原理，也利于后期维护。

2、kubeadm 部署方式介绍

kubeadm 是官方社区推出的一个用于快速部署 kubernetes 集群的工具，这个工具能通

过两条指令完成一个 kubernetes 集群的部署：

第一、创建一个

Master 节点

kubeadm init

第二， 将 Node

节点加入到当前集群中 $ kubeadm join <Master 节点的 IP 和端口 >

3、安装要求

在开始之前，部署 Kubernetes 集群机器需要满足以下几个条件：

一台或多台机器，操作系统 CentOS7.x-86_x64

硬件配置：2GB 或更多 RAM，2 个 CPU 或更多 CPU，硬盘 30GB 或更多

集群中所有机器之间网络互通

可以访问外网，需要拉取镜像

禁止 swap 分区

4、最终目标

（1） 在所有节点上安装 Docker 和 kubeadm

（2）部署 Kubernetes Master

（3）部署容器网络插件

（4）部署 Kubernetes Node，将节点加入 Kubernetes 集群中

（5）部署 Dashboard

Web 页面，可视化查看 Kubernetes 资源

 

环境准备

角色 IP

• k8s-master 01 10.11.83.57

• k8s-master 02 10.11.83.56

• k8s-master 03 10.11.83.55

• k8s-node01 10.11.83.58

• k8s-node02 10.11.83.59

• vip 10.11.83.100

• etcd-1 10.11.83.52

• etcd-1 10.11.83.53

• etcd-1 10.11.83.54

 

所有集群都执行：

[root@k8s-master1 ]# systemctl stop firewalld && systemctl disable firewalld

关闭 selinux：

[root@k8s-master1 ]# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 
[root@k8s-master1 ]# setenforce 0 # 临时

关闭 swap：

[root@k8s-master1 ]# swapoff -a # 临时 
[root@k8s-master1 ]# sed -ri 's/.*swap.*/$&/' /etc/fstab # 永久关闭
[root@k8s-master1 ]# cat /etc/fstab # 查看

修改主机名：

[root@k8s-master1 ]# hostnamectl set-hostname <hostname>

添加 hosts：

[root@k8s-master1 ]#  cat >> /etc/hosts << EOF
 10.11.83.100    master.k8s.io   k8s-vip
 10.11.83.55    master3.k8s.io k8s-master3
 10.11.83.56    master2.k8s.io k8s-master2
 10.11.83.57    master1.k8s.io k8s-master1
 10.11.83.58    node1.k8s.io k8s-node1
 10.11.83.59    node2.k8s.io k8s-node2
 10.11.83.52     etcd-1
 10.11.83.53     etcd-2
 10.11.83.54     etcd-3
EOF

将桥接的 IPv4 流量传递到 iptables 的链：

[root@k8s-master1 ]# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@k8s-master1 ]# sysctl --system # 生效

时间同步：

[root@k8s-master1 ]# yum install ntpdate -y 
[root@k8s-master1 ]# ntpdate time.windows.com

master 安装keepalived+haproxy:

[root@k8s-master1 ]# yum install -y keepalived haproxy

配置haproxy服务:

/etc/haproxy/haproxy.cfg

三个master节点的haproxy配置完全一致：

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon 
       
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------  
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#--------------------------------------------------------------------- 
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver    
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server      master1.k8s.io   10.11.83.57:6443 check
    server      master2.k8s.io   10.11.83.56:6443 check
    server      master3.k8s.io   10.11.83.55:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
    bind                 *:1080
    stats auth           admin:awesomePassword
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats

启动haproxy:

[root@k8s-master1 ]# systemctl enable haproxy && systemctl start haproxy

配置keepalived，需要注意的地方：

1、master1节点的state是MASTER，master2和master3节点的是BACKUP

2、master1的权重值priority是100，master2和master3依次递减，比如master2是90，master3是80

master1

cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state MASTER 
    interface ens192 
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    virtual_ipaddress {
        10.11.83.100
    }
    track_script {
        check_haproxy
    }

}
EOF

master2

cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens192 
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    virtual_ipaddress {
        10.11.83.100
    }
    track_script {
        check_haproxy
    }

}
EOF

master3

cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens192 
    virtual_router_id 51
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    virtual_ipaddress {
        10.11.83.100
    }
    track_script {
        check_haproxy
    }

}
EOF

启动keepalived：

[root@k8s-master1 ]# systemctl enable keepalived && systemctl start keepalived

查看master01上是否多了一个VIP地址是10.11.83.60

[root@k8s-master1 ]# ip a s ens192

2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:ab:41:00 brd ff:ff:ff:ff:ff:ff
    inet 10.11.83.52/24 brd 10.11.83.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 10.11.83.100/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feab:4100/64 scope link 
       valid_lft forever preferred_lft forever

安装docker

[root@k8s-master1 ]# yum list docker-ce.x86_64 --showduplicates | sort -r	#查看docker版本

[root@k8s-master1 ]# yum install -y wget
[root@k8s-master1 ]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
[root@k8s-master1 ]# yum -y install docker-ce-18.06.1.ce-3.el7
[root@k8s-master1 ]# systemctl enable docker && systemctl start docker
[root@k8s-master1 ]# docker --version

[root@k8s-master1 ]# cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

[root@k8s-master1 ]# systemctl restart docker

添加阿里云yum软件源:

[root@k8s-master1 ]# cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

 

安装master节点:

安装kubectl、kubeadm和kubelet，可以指定安装的版本：

[root@k8s-master1 ]# yum install -y kubeadm-1.19.2 kubectl-1.19.2 kubelet-1.19.2

配置kubelet开机自启动：

[root@k8s-master1 ]# systemctl enable kubelet

部署kubernetes master:

[root@k8s-master1 ]# mkdir /usr/local/kubernetes/manifests -p
[root@k8s-master1 ]# cd /usr/local/kubernetes/manifests/
[root@k8s-master1 manifests]# vim kubeadm-config.yaml

第一种是使用单节点etcd模式：

apiServer:
  certSANs:
    - k8s-master1
    - k8s-master2
    - k8s-master3
    - master.k8s.io
    - 10.11.83.100
    - 10.11.83.55
    - 10.11.83.56
    - 10.11.83.57
    - 127.0.0.1
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns: 
  type: CoreDNS
etcd:
  local:    
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.19.2
networking: 
  dnsDomain: cluster.local  
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.9.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"

第二种是使用集群etcd模式：

apiServer:
  certSANs:
    - k8s-master1
    - k8s-master2
    - k8s-master3
    - master.k8s.io
    - 10.11.83.100
    - 10.11.83.55
    - 10.11.83.56
    - 10.11.83.57
    - 127.0.0.1
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns: 
  type: CoreDNS
etcd:
#  local:    
#    dataDir: /var/lib/etcd
  external:
    endpoints:
    - https://10.11.83.52:2379
    - https://10.11.83.53:2379
    - https://10.11.83.54:2379
    caFile: /etc/kubernetes/pki/etcd/ca.pem
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.pem
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client-key.pem
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.19.2
networking: 
  dnsDomain: cluster.local  
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.9.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"

在vip节点执行kubeadm init 初始化:

kubeadm init --config kubeadm-config.yaml

按照提示保存以下内容，一会要使用：

 kubeadm join master.k8s.io:16443 --token 5brt32.v018jrt3izu9ttlx \
    --discovery-token-ca-cert-hash sha256:49577b267e55843780eeb7094663fe1c9485e0dfdf90e675973887707b342138 \
    --control-plane 

按照提示配置环境变量，使用kubectl工具：

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看集群状态:

kubectl get nodes
kubectl get cs
kubectl get pods -n kube-system

安装pod网络插件（CNI）:

#再master01上执行
mkdir flannel
cd flannel
vi kube-flannel.yml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.13.0-rc2
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.13.0-rc2
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

#安装flannel网络
[root@k8s-master1 flannel]# kubectl apply -f kube-flannel.yml
#检查
[root@k8s-master1 ]# kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-6d56c8448f-2g2qt              1/1     Running   0          6m43s
coredns-6d56c8448f-zrcrs              1/1     Running   0          6m43s
kube-apiserver-k8s-master1            1/1     Running   0          6m53s
kube-controller-manager-k8s-master1   1/1     Running   0          3m20s
kube-flannel-ds-xws9t                 1/1     Running   0          6s
kube-proxy-rxlr6                      1/1     Running   0          6m43s
kube-scheduler-k8s-master1            1/1     Running   0          3m38s

master节点加入集群中:

先保证主节点处于ready状态

[root@k8s-master1 ]# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
k8s-master1   Ready      master   9h    v1.19.2

在master1中执行:

ssh root@10.11.83.56 mkdir -p /etc/kubernetes/pki/etcd
ssh root@10.11.83.55 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@10.11.83.56:/etc/kubernetes
scp /etc/kubernetes/admin.conf root@10.11.83.55:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*,apiserver-etcd-client.*} root@10.11.83.56:/etc/kubernetes/pki
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*,apiserver-etcd-client.*} root@10.11.83.55:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@10.11.83.56:/etc/kubernetes/pki/etcd
scp /etc/kubernetes/pki/etcd/ca.* root@10.11.83.55:/etc/kubernetes/pki/etcd

在master3和master3上执行刚才保存的语句，带上参数--control-plane表示把master控制节点加入集群

  kubeadm join master.k8s.io:16443 --token cuvixy.cc6x9w7q7dxmculc \
    --discovery-token-ca-cert-hash sha256:9443818d45fbe6ac03ae6938922e634708281c6a38a072aee50aa6b76a5c526b \
    --control-plane 

在master2和master3上执行:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看集群状态:

[root@k8s-master1 ]# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
k8s-master1   Ready    master   18m     v1.19.2
k8s-master2   Ready    master   2m49s   v1.19.2
k8s-master3   Ready    master   2m28s   v1.19.2

[root@k8s-master1 ]# kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-6d56c8448f-2g2qt              1/1     Running   0          18m
coredns-6d56c8448f-zrcrs              1/1     Running   0          18m
kube-apiserver-k8s-master1            1/1     Running   0          18m
kube-apiserver-k8s-master2            1/1     Running   0          2m54s
kube-apiserver-k8s-master3            1/1     Running   0          2m33s
kube-controller-manager-k8s-master1   1/1     Running   0          15m
kube-controller-manager-k8s-master2   1/1     Running   0          2m56s
kube-controller-manager-k8s-master3   1/1     Running   0          2m35s
kube-flannel-ds-66mzc                 1/1     Running   0          2m56s
kube-flannel-ds-dvzjr                 1/1     Running   0          2m35s
kube-flannel-ds-xws9t                 1/1     Running   0          12m
kube-proxy-9lgck                      1/1     Running   0          2m56s
kube-proxy-gm2qn                      1/1     Running   0          2m35s
kube-proxy-rxlr6                      1/1     Running   0          18m
kube-scheduler-k8s-master1            1/1     Running   0          15m
kube-scheduler-k8s-master2            1/1     Running   0          2m55s
kube-scheduler-k8s-master3            1/1     Running   0          2m34s

至此，k8s master集群搭建成功!!!

 

添加node节点:

node节点执行：注意这里去掉了--control-plane参数

 kubeadm join master.k8s.io:16443 --token cuvixy.cc6x9w7q7dxmculc \
    --discovery-token-ca-cert-hash sha256:9443818d45fbe6ac03ae6938922e634708281c6a38a072aee50aa6b76a5c526b

集群网络重新安装:master1中执行：

kubectl delete -f kube-flannel.yml
kubectl apply -f kube-flannel.yml

查看所有的pods:

[root@k8s-master1 ]# kubectl get pods --all-namespaces
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-6d56c8448f-2g2qt              1/1     Running   0          25m
coredns-6d56c8448f-zrcrs              1/1     Running   0          25m
kube-apiserver-k8s-master1            1/1     Running   0          25m
kube-apiserver-k8s-master2            1/1     Running   0          9m46s
kube-apiserver-k8s-master3            1/1     Running   0          9m25s
kube-controller-manager-k8s-master1   1/1     Running   0          22m
kube-controller-manager-k8s-master2   1/1     Running   0          9m48s
kube-controller-manager-k8s-master3   1/1     Running   0          9m27s
kube-flannel-ds-66mzc                 1/1     Running   0          9m48s
kube-flannel-ds-dvzjr                 1/1     Running   0          9m27s
kube-flannel-ds-pwgxt                 1/1     Running   0          70s
kube-flannel-ds-rkfgz                 1/1     Running   0          81s
kube-flannel-ds-xws9t                 1/1     Running   0          18m
kube-proxy-4fnp9                      1/1     Running   0          81s
kube-proxy-8vzrq                      1/1     Running   0          70s
kube-proxy-9lgck                      1/1     Running   0          9m48s
kube-proxy-gm2qn                      1/1     Running   0          9m27s
kube-proxy-rxlr6                      1/1     Running   0          25m
kube-scheduler-k8s-master1            1/1     Running   0          22m
kube-scheduler-k8s-master2            1/1     Running   0          9m47s
kube-scheduler-k8s-master3            1/1     Running   0          9m26s

检查k8s集群状态：

[root@k8s-master1 ]# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
k8s-master1   Ready    master   24m     v1.19.2
k8s-master2   Ready    master   8m50s   v1.19.2
k8s-master3   Ready    master   8m29s   v1.19.2
k8s-node1     Ready    <none>   23s     v1.19.2
k8s-node2     Ready    <none>   12s     v1.19.2

至此，kukernetes高可用集群搭建和测试完毕！