kubeadm High availability cluster(1.23)
1、部署环境说明
本文通过kubeadm搭建一个高可用的k8s集群，kubeadm可以帮助我们快速的搭建k8s集群，高可用主要体现在对master节点组件及etcd存储的高可用，文中使用到的服务器ip及角色对应如下：
| 主机名称 | ip地址 | 角色 | 配置 |
|---|---|---|---|
| - | 192.168.182.10 | 虚拟ip(vip) | |
| k8s-master-01 | 192.168.182.11 | master | 8c/8g |
| K8s-master-02 | 192.168.182.12 | master | 8c/8g |
| K8s-master-03 | 192.168.182.13 | master | 8c/8g |
| k8s-node-01 | 192.168.182.14 | node | 8c/8g |
| K8s-node-02 | 192.168.182.15 | node | 8c/8g |
| K8s-node-03 | 192.168.182.16 | node | 8c/8g |
2、集群架构及部署准备工作
2.1、集群架构说明
前面提到高可用主要体现在master相关组件及etcd，master中apiserver是集群的入口，搭建三个master通过keepalived提供一个vip实现高可用，并且添加haproxy来为apiserver提供反向代理的作用，这样来自haproxy的所有请求都将轮询转发到后端的master节点上。如果仅仅使用keepalived，当集群正常工作时，所有流量还是会到具有vip的那台master上，因此加上了haproxy使整个集群的master都能参与进来，集群的健壮性更强。对应架构图如下所示：
2.2、修改hosts及hostname
所有节点修改主机名和hosts文件
cat >> /etc/hosts <<EOF
192.168.182.10 master.k8s.io k8s-vip
192.168.182.11 master01.k8s.io k8s-master-01
192.168.182.12 master02.k8s.io k8s-master-02
192.168.182.13 master03.k8s.io k8s-master-03
192.168.182.14 node01.k8s.io k8s-node-01
192.168.182.15 node02.k8s.io k8s-node-02
192.168.182.16 node03.k8s.io k8s-node-03
EOF
2.3、其他准备
所有节点操作
- 修改内核
# vi /etc/default/grub添加cgroup.memory=nokmem字段
GRUB_CMDLINE_LINUX="crashkernel=auto net.ifnames=0 biosdevname=0 intel_pstate=disable cgroup.memory=nokmem"
# 生成配置：
/usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
# 重启机器：
reboot
# 验证：
cat /sys/fs/cgroup/memory/kubepods/burstable/pod*/*/memory.kmem.slabinfo 无输出即可。
- 主机时间同步
时间同步可以通过chrony或者ntp来实现
#查看时区
timedatectl
#检查chronyd状态
systemctl status chronyd
#添加时间服务器
vi /etc/chrony.conf
server 你要添加的时间服务器的ip iburst
-
关闭防火墙和selinux
systemctl disable firewalld --now && setenforce 0 && sed -i 's@SELINUX=enforcing@SELINUX=disabled@g' /etc/selinux/config -
进行ssh免密
# 生成公私钥对 [root@k8s-master-01 ~]# ssh-keygen 一路回车 # 发送公私钥对到各节点 [root@k8s-master-01 ~]# ssh-copy-id k8s-master-01 && ssh-copy-id k8s-master-02 && ssh-copy-id k8s-master-03 && ssh-copy-id k8s-node-01 && ssh-copy-id k8s-node-02 && ssh-copy-id k8s-node-03 -
禁用swap
kubeadm会检查当前主机是否禁用了swap，如果启动了swap将导致安装不能正常进行，所以需要禁用所有的swap。
# 临时关闭
$ swapoff -a && sysctl -w vm.swappiness=0
# 永久关闭，在文件中添加注释
$ vim /etc/fstab
...
UUID=7bf41652-e6e9-415c-8dd9-e112641b220e /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0
# 或者利用sed命令完事儿
$ sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
- 设置系统其它参数
开启路由转发
$ cat >> /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ modprobe br_netfilter
$ sysctl -p /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
设置资源配置文件
cat >> /etc/security/limits.conf << EOF
* hard nofile 65535
* soft nofile 65535
* hard nproc 65535
* soft nproc 65535
* soft memlock unlimited
* hard memlock unlimited
EOF
- 安装相关包
$ yum install -y conntrack-tools libseccomp libtool-ltdl
3、部署keepalived
在三台master操作
3.1、安装
$ yum install -y keepalived
3.2、配置
默认的keepalived配置较复杂，这里用更为简明的方式进行配置，另外的两台master配置和上面类似，只需要修改对应的state配置为BACKUP，priority权重值不同即可，配置中的其他字段这里不做说明。
k8s-master-01的配置：
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 250
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.182.10
}
track_script {
check_haproxy
}
}
EOF
k8s-master-02的配置：
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.182.10
}
track_script {
check_haproxy
}
}
EOF
k8s-master-03的配置：
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.182.10
}
track_script {
check_haproxy
}
}
EOF
3.3、启动和检查
在三台master节点都启动服务
# 设置开机启动
$ systemctl enable keepalived.service
# 启动keepalived
$ systemctl start keepalived.service
# 查看启动状态
$ systemctl status keepalived.service
启动后查看k8s-master-01的网卡信息
[root@k8s-master-01 ~]# ip a s ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:d9:b6:6a brd ff:ff:ff:ff:ff:ff
inet 192.168.182.11/24 brd 192.168.182.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.182.10/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::af83:7b83:9859:d413/64 scope link noprefixroute
valid_lft forever preferred_lft forever
尝试停掉k8s-master-01的keepalived服务，查看vip是否能漂移到其他的master，并且重新启动k8s-master-01的keepalived服务，查看vip是否能正常漂移回来，证明配置没有问题。
4、部署haproxy
在三台master操作
4.1、安装
$ yum install -y haproxy
4.2、配置
三台master节点的配置均相同，配置中声明了后端代理的三个master节点服务器，指定了haproxy运行的端口为16443等，因此16443端口为集群的入口，其他的配置不做赘述。
cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master01.k8s.io 192.168.182.11:6443 check
server master02.k8s.io 192.168.182.12:6443 check
server master03.k8s.io 192.168.182.13:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:1080
stats auth admin:awesomePassword
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
EOF
4.3、启动和检查
在三台master节点都启动服务
# 设置开机启动
$ systemctl enable haproxy
# 开启haproxy
$ systemctl start haproxy
# 查看启动状态
$ systemctl status haproxy
检查端口
[root@k8s-master-01 ~]# netstat -lntup|grep haproxy
tcp 0 0 0.0.0.0:1080 0.0.0.0:* LISTEN 7067/haproxy
tcp 0 0 0.0.0.0:16443 0.0.0.0:* LISTEN 7067/haproxy
udp 0 0 0.0.0.0:47041 0.0.0.0:* 7066/haproxy
5、安装docker
所有节点操作，使用yum安装，参考安装 Docker 并配置镜像加速源
5.1、安装
# step 1: 安装必要的一些系统工具
$ yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
$ sudo yum-config-manager --add-repo https://mirrors.cloud.tencent.com/docker-ce/linux/centos/docker-ce.repo
# Step 3: 查找Docker-CE的版本:
$ yum list docker-ce.x86_64 --showduplicates | sort -r
# Step 4: 安装指定版本的Docker-CE
$ yum makecache fast
$ yum install -y docker-ce-20.10.12
5.2、配置
修改docker的配置文件，目前k8s推荐使用的docker文件驱动是systemd，按照k8s官方文档可查看如何配置
cat <<EOF > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
修改docker的服务配置文件，指定docker的数据目录为外挂的磁盘--graph /data/docker
$ vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --graph /data/docker
5.3、启动
启动docker服务
$ systemctl daemon-reload
$ systemctl start docker.service
$ systemctl enable docker.service
$ systemctl status docker.service
检查docker信息
$ docker version
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:41 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:44:05 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
6、安装kubeadm，kubelet和kubectl
所有节点操作
6.1、添加阿里云k8s的yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
添加之后运行命令init 6重启使yum生效
6.2、安装
$ yum install -y kubelet-1.23.3 kubeadm-1.23.3 kubectl-1.23.3
$ systemctl enable kubelet
6.3、配置kubectl自动补全
[root@k8s-master-01 ~]# source <(kubectl completion bash)
[root@k8s-master-01 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
7、安装master
在具有vip的master上操作，这里为k8s-master-01
7.1、创建kubeadm配置文件
[root@k8s-master-01 ~]# mkdir /usr/local/kubernetes/manifests -p
[root@k8s-master-01 ~]# cd /usr/local/kubernetes/manifests/
[root@k8s-master-01 manifests]# vim kubeadm-config.yaml
apiServer:
certSANs:
- k8s-master-01
- k8s-master-02
- k8s-master-03
- master.k8s.io
- 192.168.182.10
- 192.168.182.11
- 192.168.182.12
- 192.168.182.13
- 127.0.0.1
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.23.3
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.1.0.0/16
scheduler: {}
初始化master节点
[root@k8s-master-01 manifests]# kubeadm init --config kubeadm-config.yaml
your configuration file uses an old API spec: "kubeadm.k8s.io/v1beta1". Please use kubeadm v1.15 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
To see the stack trace of this error execute with --v=5 or higher
[root@master1 manifests]# kubeadm init --config kubeadm-config.yaml higher
unknown command "higher" for "kubeadm init"
[root@master1 manifests]# kubeadm init --config kubeadm-config.yaml
[init] Using Kubernetes version: v1.23.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-01 k8s-master-02 k8s-master-03 k8svip kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master.k8s.io master1] and IPs [10.1.0.1 192.168.182.11 192.168.182.10 192.168.182.12 192.168.182.13 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master1] and IPs [192.168.182.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master1] and IPs [192.168.182.11 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.016472 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ib8ffm.6kzhauayqi1mx33q
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join k8svip:16443 --token ib8ffm.6kzhauayqi1mx33q \
--discovery-token-ca-cert-hash sha256:6a8e5c64b7aa00d6f5d57c0393d5f39c992d5a7ae18f2827c0dc93bc777040b0 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8svip:16443 --token ib8ffm.6kzhauayqi1mx33q \
--discovery-token-ca-cert-hash sha256:6a8e5c64b7aa00d6f5d57c0393d5f39c992d5a7ae18f2827c0dc93bc777040b0
7.3、按照提示配置环境变量
[root@k8s-master-01 manifests]# mkdir -p $HOME/.kube
[root@k8s-master-01 manifests]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master-01 manifests]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
7.4、查看集群状态
[root@k8s-master-01 manifests]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
[root@k8s-master-01 manifests]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-58cc8c89f4-56n7g 0/1 Pending 0 87s
coredns-58cc8c89f4-zclz7 0/1 Pending 0 87s
etcd-k8s-master-01 1/1 Running 0 18s
kube-apiserver-k8s-master-01 1/1 Running 0 21s
kube-controller-manager-k8s-master-01 1/1 Running 0 33s
kube-proxy-ptjjn 1/1 Running 0 87s
kube-scheduler-k8s-master-01 1/1 Running 0 25s
8、安装集群网络
master节点操作
8.1、获取yaml
从官方地址获取到flannel的yaml
[root@k8s-master-01 manifests]# mkdir flannel
[root@k8s-master-01 manifests]# cd flannel
[root@k8s-master-01 flannel]# wget -c https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
8.2、安装
[root@k8s-master-01 flannel]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
8.3、检查
[root@k8s-master-01 flannel]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-58cc8c89f4-56n7g 1/1 Running 0 20m
coredns-58cc8c89f4-zclz7 1/1 Running 0 20m
etcd-k8s-master-01 1/1 Running 0 19m
kube-apiserver-k8s-master-01 1/1 Running 0 19m
kube-controller-manager-k8s-master-01 1/1 Running 0 19m
kube-flannel-ds-amd64-8d8bc 1/1 Running 0 51s
kube-proxy-ptjjn 1/1 Running 0 20m
kube-scheduler-k8s-master-01 1/1 Running 0 19m
9、其他节点加入集群
9.1、master加入集群
9.1.1、复制密钥及相关文件
在第一次执行init的机器，此处为k8s-master-01上操作
复制文件到k8s-master-02
[root@k8s-master-01 ~]# ssh root@192.168.182.12 mkdir -p /etc/kubernetes/pki/etcd
[root@k8s-master-01 ~]# scp /etc/kubernetes/admin.conf root@192.168.182.12:/etc/kubernetes
admin.conf 100% 5454 465.7KB/s 00:00
[root@k8s-master-01 ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@192.168.182.12:/etc/kubernetes/pki
ca.crt 100% 1025 89.2KB/s 00:00
ca.key 100% 1675 212.1KB/s 00:00
sa.key 100% 1679 210.1KB/s 00:00
sa.pub 100% 451 56.5KB/s 00:00
front-proxy-ca.crt 100% 1038 131.9KB/s 00:00
front-proxy-ca.key 100% 1679 208.3KB/s 00:00
[root@k8s-master-01 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@192.168.182.12:/etc/kubernetes/pki/etcd
ca.crt 100% 1017 138.8KB/s 00:00
ca.key
复制文件到k8s-master-03
[root@k8s-master-01 ~]# ssh root@192.168.182.13 mkdir -p /etc/kubernetes/pki/etcd
[root@k8s-master-01 ~]# scp /etc/kubernetes/admin.conf root@192.168.182.13:/etc/kubernetes
admin.conf 100% 5454 824.2KB/s 00:00
[root@k8s-master-01 ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@192.168.182.13:/etc/kubernetes/pki
ca.crt 100% 1025 144.6KB/s 00:00
ca.key 100% 1675 218.0KB/s 00:00
sa.key 100% 1679 245.7KB/s 00:00
sa.pub 100% 451 57.3KB/s 00:00
front-proxy-ca.crt 100% 1038 132.6KB/s 00:00
front-proxy-ca.key 100% 1679 213.4KB/s 00:00
[root@k8s-master-01 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@192.168.182.13:/etc/kubernetes/pki/etcd
ca.crt 100% 1017 55.0KB/s 00:00
ca.key
9.1.2、master加入集群
分别在其他两台master上操作，执行在k8s-master-01上init后输出的join命令，如果找不到了，可以在master01上执行以下命令输出
[root@k8s-master-01 ~]# kubeadm token create --print-join-command
kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba
在k8s-master-02上执行join命令，需要带上参数--control-plane表示把master控制节点加入集群
[root@k8s-master-02 ~]# kubeadm join master.k8s.io:16443 --token bottw9.qd4xhxik09vjmj71 --discovery-token-ca-cert-hash sha256:e89543d0125b2cf2cb7bd5675805b7eac22ea4f42bfbcfe1b49811f43d1d24bb --control-plane
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-02 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master.k8s.io] and IPs [10.1.0.1 192.168.182.12]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-02 localhost] and IPs [192.168.182.12 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-02 localhost] and IPs [192.168.182.12 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node k8s-master-02 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master-02 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
[root@k8s-master-02 ~]# mkdir -p $HOME/.kube
[root@k8s-master-02 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master-02 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
同样的，在k8s-master-03上执行join命令，输出及后续相关的步骤同上
[root@k8s-master-03 ~]# kubeadm join master.k8s.io:16443 --token bottw9.qd4xhxik09vjmj71 \
--discovery-token-ca-cert-hash sha256:e89543d0125b2cf2cb7bd5675805b7eac22ea4f42bfbcfe1b49811f43d
--control-plane
[root@k8s-master-03 ~]# mkdir -p $HOME/.kube
[root@k8s-master-03 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master-03 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
9.1.3、检查
在其中一台master上执行命令检查集群及pod状态
[root@k8s-master-01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master-01 Ready master 36m v1.23.2
k8s-master-02 Ready master 3m20s v1.23.2
k8s-master-03 Ready master 21s v1.23.2
[root@k8s-master-01 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-56n7g 1/1 Running 0 36m
kube-system coredns-58cc8c89f4-zclz7 1/1 Running 0 36m
kube-system etcd-k8s-master-01 1/1 Running 0 35m
kube-system etcd-k8s-master-02 1/1 Running 0 3m55s
kube-system etcd-k8s-master-03 1/1 Running 0 56s
kube-system kube-apiserver-k8s-master-01 1/1 Running 0 35m
kube-system kube-apiserver-k8s-master-02 1/1 Running 0 3m55s
kube-system kube-apiserver-k8s-master-03 1/1 Running 0 57s
kube-system kube-controller-manager-k8s-master-01 1/1 Running 1 35m
kube-system kube-controller-manager-k8s-master-02 1/1 Running 0 3m55s
kube-system kube-controller-manager-k8s-master-03 1/1 Running 0 57s
kube-system kube-flannel-ds-amd64-7hnhl 1/1 Running 1 3m56s
kube-system kube-flannel-ds-amd64-8d8bc 1/1 Running 0 17m
kube-system kube-flannel-ds-amd64-fp2rb 1/1 Running 0 57s
kube-system kube-proxy-gzswt 1/1 Running 0 3m56s
kube-system kube-proxy-hdrq7 1/1 Running 0 57s
kube-system kube-proxy-ptjjn 1/1 Running 0 36m
kube-system kube-scheduler-k8s-master-01 1/1 Running 1 35m
kube-system kube-scheduler-k8s-master-02 1/1 Running 0 3m55s
kube-system kube-scheduler-k8s-master-03 1/1 Running 0 57s
9.2、node加入集群
9.2.1、node加入集群
分别在其他三台node节点上操作，执行join命令
在k8s-node-01上操作
[root@k8s-node-02 ~]# kubeadm join master.k8s.io:16443 --token bottw9.qd4xhxik09vjmj71 \
--discovery-token-ca-cert-hash sha256:e89543d0125b2cf2cb7bd5675805b7eac22ea4f42bfbcfe1b49811f43
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
同理
[root@k8s-node-02 ~]# kubeadm join master.k8s.io:16443 --token bottw9.qd4xhxik09vjmj71 \
--discovery-token-ca-cert-hash sha256:e89543d0125b2cf2cb7bd5675805b7eac22ea4f42bfbcfe1b49811f43
[root@k8s-node-03 ~]# kubeadm join master.k8s.io:16443 --token bottw9.qd4xhxik09vjmj71 \
--discovery-token-ca-cert-hash sha256:e89543d0125b2cf2cb7bd5675805b7eac22ea4f42bfbcfe1b49811f43
9.2.2、检查
[root@k8s-master-01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master-01 Ready master 42m v1.23.2
k8s-master-02 Ready master 9m3s v1.23.2
k8s-master-03 Ready master 6m4s v1.23.2
k8s-node-01 Ready <none> 31s v1.23.2
k8s-node-02 Ready <none> 28s v1.23.2
k8s-node-03 Ready <none> 38s v1.23.2
[root@k8s-master-01 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-56n7g 1/1 Running 0 41m
kube-system coredns-58cc8c89f4-zclz7 1/1 Running 0 41m
kube-system etcd-k8s-master-01 1/1 Running 0 40m
kube-system etcd-k8s-master-02 1/1 Running 0 9m4s
kube-system etcd-k8s-master-03 1/1 Running 0 6m5s
kube-system kube-apiserver-k8s-master-01 1/1 Running 0 40m
kube-system kube-apiserver-k8s-master-02 1/1 Running 0 9m4s
kube-system kube-apiserver-k8s-master-03 1/1 Running 0 6m6s
kube-system kube-controller-manager-k8s-master-01 1/1 Running 1 40m
kube-system kube-controller-manager-k8s-master-02 1/1 Running 0 9m4s
kube-system kube-controller-manager-k8s-master-03 1/1 Running 0 6m6s
kube-system kube-flannel-ds-amd64-7hnhl 1/1 Running 1 9m5s
kube-system kube-flannel-ds-amd64-8d8bc 1/1 Running 0 22m
kube-system kube-flannel-ds-amd64-bwwlx 1/1 Running 0 33s
kube-system kube-flannel-ds-amd64-fp2rb 1/1 Running 0 6m6s
kube-system kube-flannel-ds-amd64-g9vdj 1/1 Running 0 40s
kube-system kube-flannel-ds-amd64-xcbfr 1/1 Running 0 30s
kube-system kube-proxy-485dl 1/1 Running 0 30s
kube-system kube-proxy-8p688 1/1 Running 0 40s
kube-system kube-proxy-fdq7c 1/1 Running 0 33s
kube-system kube-proxy-gzswt 1/1 Running 0 9m5s
kube-system kube-proxy-hdrq7 1/1 Running 0 6m6s
kube-system kube-proxy-ptjjn 1/1 Running 0 41m
kube-system kube-scheduler-k8s-master-01 1/1 Running 1 40m
kube-system kube-scheduler-k8s-master-02 1/1 Running 0 9m4s
kube-system kube-scheduler-k8s-master-03 1/1 Running 0 6m6s
10、创建并了解一个 nginx Deployment
10.1、创建 Deployment
你可以通过创建一个 Kubernetes Deployment 对象来运行一个应用, 且你可以在一个 YAML 文件中描述 Deployment。例如, 下面这个 YAML 文件描述了一个运行 nginx:1.14.2 Docker 镜像的 Deployment：
application/deployment.yaml 
#api版本
apiVersion: apps/v1
#类型:部署组
kind: Deployment
#元数据：
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
#容器镜像
image: nginx:1.14.2
ports:
#容器端口
- containerPort: 80
-
通过 YAML 文件创建一个 Deployment：
kubectl apply -f https://k8s.io/examples/application/deployment.yaml -
显示 Deployment 相关信息：
kubectl describe deployment nginx-deployment输出类似于这样：
Name: nginx-deployment Namespace: default CreationTimestamp: Tue, 30 Aug 2016 18:11:37 -0700 Labels: app=nginx Annotations: deployment.kubernetes.io/revision=1 Selector: app=nginx Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 1 max unavailable, 1 max surge Pod Template: Labels: app=nginx Containers: nginx: Image: nginx:1.7.9 Port: 80/TCP Environment: <none> Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: nginx-deployment-1771418926 (2/2 replicas created) No events. -
列出 Deployment 创建的 Pods：
kubectl get pods -l app=nginx输出类似于这样：
NAME READY STATUS RESTARTS AGE nginx-deployment-1771418926-7o5ns 1/1 Running 0 16h nginx-deployment-1771418926-r18az 1/1 Running 0 16h -
展示某一个 Pod 信息：
kubectl describe pod <pod-name>这里的
<pod-name>是某一 Pod 的名称。
10.2、更新 Deployment
你可以通过更新一个新的 YAML 文件来更新 Deployment。下面的 YAML 文件指定该 Deployment 镜像更新为 nginx 1.16.1。
application/deployment-update.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.16.1 # Update the version of nginx from 1.14.2 to 1.16.1
ports:
- containerPort: 80
-
应用新的 YAML：
kubectl apply -f https://k8s.io/examples/application/deployment-update.yaml -
查看该 Deployment 以新的名称创建 Pods 同时删除旧的 Pods：
kubectl get pods -l app=nginx
10.3、通过增加副本数来扩缩应用
你可以通过应用新的 YAML 文件来增加 Deployment 中 Pods 的数量。 下面的 YAML 文件将 replicas 设置为 4，指定该 Deployment 应有 4 个 Pods：
application/deployment-scale.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 4 # Update the replicas from 2 to 4
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
-
应用新的 YAML 文件：
kubectl apply -f https://k8s.io/examples/application/deployment-scale.yaml -
验证 Deployment 有 4 个 Pods：
kubectl get pods -l app=nginx输出的结果类似于:
NAME READY STATUS RESTARTS AGE nginx-deployment-148880595-4zdqq 1/1 Running 0 25s nginx-deployment-148880595-6zgi1 1/1 Running 0 25s nginx-deployment-148880595-fxcez 1/1 Running 0 2m nginx-deployment-148880595-rwovn 1/1 Running 0 2m
10.4、删除 Deployment
基于名称删除 Deployment：
kubectl delete deployment nginx-deployment
11、安装dashboard
11.1、部署dashboard
地址：https://github.com/kubernetes/dashboard
文档：https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
部署最新版本v2.5.0，下载yaml
[root@k8s-master-01 manifests]# cd /usr/local/kubernetes/manifests/
[root@k8s-master-01 manifests]# mkdir dashboard
[root@k8s-master-01 manifests]# cd dashboard/
[root@k8s-master-01 dashboard]# wget -c https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml
# 修改service类型为nodeport
[root@k8s-master-01 dashboard]# vim recommended.yaml
...
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
...
[root@k8s-master-01 dashboard]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard unchanged
serviceaccount/kubernetes-dashboard unchanged
service/kubernetes-dashboard unchanged
secret/kubernetes-dashboard-certs unchanged
secret/kubernetes-dashboard-csrf configured
secret/kubernetes-dashboard-key-holder unchanged
configmap/kubernetes-dashboard-settings unchanged
role.rbac.authorization.k8s.io/kubernetes-dashboard unchanged
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard unchanged
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard unchanged
deployment.apps/kubernetes-dashboard unchanged
service/dashboard-metrics-scraper unchanged
deployment.apps/dashboard-metrics-scraper unchanged
[root@k8s-master-01 dashboard]# kubectl get pods -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-76585494d8-62vp9 1/1 Running 0 6m47s
kubernetes-dashboard-b65488c4-5t57x 1/1 Running 0 6m48s
[root@k8s-master-01 dashboard]# kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.1.207.27 <none> 8000/TCP 7m6s
kubernetes-dashboard NodePort 10.1.207.168 <none> 443:30001/TCP 7m7s
# 在node上通过https://nodeip:30001访问是否正常
11.2、创建service account并绑定默认cluster-admin管理员集群角色
[root@k8s-master-01 dashboard]# vim dashboard-adminuser.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
[root@k8s-master-01 dashboard]# kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
获取token
[root@k8s-master-01 dashboard]# kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
[root@k8s-master-01 dashboard]# kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
Name: admin-user-token-vrbc4
Namespace: kubernetes-dashboard
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: c2c5d076-a7a5-4a1e-ae33-e8efd68da6e7
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1099 bytes
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjBFUjh0WUxLZEQwazEzdnZNUlpsRzRKTWg5Zk5VcjN4N09CSXBnRVZIeUkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXZyYmM0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJjMmM1ZDA3Ni1hN2E1LTRhMWUtYWUzMy1lOGVmZDY4ZGE2ZTciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.FeNxOw_zW9AYMINRehz-Z3DKk-mEos3_pRcTQAHsb0J7sgdUh22n_P3-SgDnt2bgh7fbOW2MeQhpFclg3GKbxIqPx5V4Oh3cOn8fgJhmrsOJWP67na7qFAYfeB8r7-avWL1T185Y7Zcodij4-GdjrsGvsy-bmYoBLF2Dtg_pIhtIb4MQuGq0IaKlJABjOry9WII2J03CZ1pCgVP_jyfkRbchxv86IV7LWjVwI5oghTe2uzhgwtIT3OJ0x6aUFc621e7nAZ3VHK2rhNAlmIc_4jmIGgmDYZdktvcRVZksjQbwHtGh-rEyW3D9mQRxlyyMyc2SC0CE4f1lEb4zAIIWNg
原文链接：https://icode9.com/content-4-1229391.html
本机器访问
master机器输入命令，此时命令为挂起状态
kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'&
在本机浏览器输入（注意必须是 http），对，没错就是这么长的连接：
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
外部机器访问
端口转发模式：
监听所有IP地址，并将8080转发至443https端口访问。
kubectl port-forward -n kubernetes-dashboard --address 0.0.0.0 service/kubernetes-dashboard 8080:443
这时在外部机器浏览器输入，（注意必须是 https），对，没错就是这么短的连接即可访问：
https://192.168.182.14:8080/
11.3、使用token登录到dashboard界面
