Kubeadm安装Kubernetes高可用集群

一、首先配置k8s(master)主节点和node节点修改/etc/hosts配置文件如下：

168.3.123 k8s-master1
168.3.124 k8s-master2
168.3.125 k8s-master3
 
168.3.128 k8s-vip  #如果不是高可用集群，该IP为k8s-master1的IP地址
 
168.3.126 node-1
168.3.127 node-2

二、所有k8s-master节点和node节点安装yum源如下

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
 
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
 
yum clean all 
yum makecache
 
yum -y install yum-utils device-mapper-persistent-data lvm2 wget psmisc net-tools telnet git
 
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
 
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

　　yum 报错问题和解决问题如下

　　1、报错

解决报错问题：

修改/etc/yum.repos.d/kubernetes.repo 里面的repo_gpgcheck=1 改为repo_gpgcheck=0

三、所有k8s-master节点和node节点关闭所有节点防火墙，selinux NetworkManager swap 服务配置

# 关闭防火墙
 
systemctl disable firewalld
systemctl stop firewalld
 
systemctl disable NetworkManager
systemctl stop NetworkManager
 
# 关闭selinux
# 临时禁用selinux
 
setenforce 0
 
# 永久关闭 修改/etc/sysconfig/selinux文件设置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
 
 
# 禁用交换分区
swapoff -a
# 永久禁用，打开/etc/fstab注释掉swap那一行。
sed -i 's/.*swap.*/#&/' /etc/fstab

四、所有节点配置limit

#临时生效
ulimit -SHn 65535
  
#永久生效
vim /etc/security/limits.conf
 
*   soft    nofile  655360
*   hard    nofile  131072
*   soft    nproc   655350
*   hard    nproc   655350
*   soft    memlock unlimited
*   hard    memlock unlimited

五、k8s-master1(主master-1)节点上面登陆其它节点，安装过程中生成配置文件和证书均在k8s-master1(主master-1)上操作，集群管理也在k8s-master1上操作。阿里云或其它云服务器上需要单独的一台kubectl服务器。密钥配置如下

ssh-keygen -t rsa
 
for i in k8s-master1 k8s-master2 k8s-master3 node-1 node-2;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

六、下载安装所有的源码文件

1	`cd` `/root; git clone https://github.com/dotbalo/k8s-ha-install.git`

七、所有节点升级系统并重启

1	`yum update -y --exclude=kernel* && reboot`

八、内核升级，本地升级内核版本为4.19

　　1、在k8s-master1节点下载内核

#下载到root目录下面
cd /root
 
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
 
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

　　2、从k8s-master1节点传到其它节点/root目录下面

1	`for` `i` `in` `k8s-master2 k8s-master3 node-1 node-2;` `do` `scp` `kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/;` `done`

　　3、所有节点安装内核

1	`cd` `/root` `&& yum localinstall -y kernel-ml-*`

　　4、所有节点更改内核启动顺序(注：保留原来内核防止升级内核升级不能启动可用修改为原来内核)

[root@k8s-master1 ~]# grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-4.19.12-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-159c5700b90c473598c0d0d88f656997
Found initrd image: /boot/initramfs-0-rescue-159c5700b90c473598c0d0d88f656997.img
done
 
[root@k8s-master1 ~]# grubby --args="user namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

　　5、检查节点内核是不是4.19

1 2	`[root@k8s-master1 ~]# grubby --default-kernel` `/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64`

　　6、所有节点重启reboot，检查内核是不是4.19

1 2	`[root@k8s-master1 ~]# uname -a` `Linux k8s-master1 4.19.12-1.el7.elrepo.x86_64` `#1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux`

九、所有节点安装ipvsadm

　　1、安装ipvsadm

1	`yum -y` `install` `ipvsadm ipset sysstat conntrack libseccomp`

　　2、所有节点配置ipvs模块，内核4.19+版本nf_conntrack_ipv4 已经改为nf_conntrack。4.18以下的版本内核使用nf_conntrack_ipv4 即可

[root@k8s-master1 ~]# vim /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

　　1）然后执行systemctl enable --now systemd-modules-load.service 命令即可。检查是否加载(注：需要重启系统才会生效)

　　3、开启k8s集群中内核参数，所有节点配置k8s内核(注：需要重启系统reboot)　　

cat <<EOF > /etc/sysctl.d/k8s.conf
vm.panic_on_oom=0
vm.overcommit_memory=1
 
fs.file-max=52706963
fs.nr_open=52706963
fs.may_detach_mounts = 1
fs.inotify.max_user_watches=89100
 
net.ipv4.ip_forward = 1
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.ipv4.conf.all.route_localnet = 1
net.bridge.bridge-nf-call-iptables = 1
 
net.bridge.bridge-nf-call-ip6tables = 1
net.core.somaxconn = 16384
net.netfilter.nf_conntrack_max=2310720
EOF

　　4、所有节点配置后，重启服务器，保证重启后内核依据加载　　

[root@k8s-master1 ~]# lsmod |grep --color=auto -e ip_vs -e nf_conntrack
ip_vs_ftp              16384  0 
nf_nat                 32768  1 ip_vs_ftp
ip_vs_sed              16384  0 
ip_vs_nq               16384  0 
ip_vs_fo               16384  0 
ip_vs_sh               16384  0 
ip_vs_dh               16384  0 
ip_vs_lblcr            16384  0 
ip_vs_lblc             16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  0 
ip_vs_wlc              16384  0 
ip_vs_lc               16384  0 
ip_vs                 151552  24 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack          143360  2 nf_nat,ip_vs
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  4 nf_conntrack,nf_nat,xfs,ip_vs

　　5、相关内核一些参数说明

net.ipv4.ip_forward=1 # 其值为0,说明禁止进行IP转发；如果是1,则说明IP转发功能已经打开。
net.ipv4.tcp_keepalive_time=600 #此参数表示TCP发送keepalive探测消息的间隔时间(秒) 
net.ipv4.tcp_keepalive_probes=3  #tcp检查次数（如果对方不予应答，探测包的发送次数）
net.ipv4.tcp_keepalive_intvl=15 #tcp检查间隔时间（keepalive探测包的发送间隔） 
net.ipv4.tcp_max_tw_buckets=36000 #配置服务器 TIME_WAIT 数量
net.ipv4.tcp_tw_reuse = 1 表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；
net.ipv4.tcp_max_orphans=327680
net.ipv4.tcp_orphans_retries=3
net.ipv4.tcp_syncookies=1 #此参数应该设置为1，防止SYN Flood
net.ipv4.tcp_max_syn_backlog = 16384 # 第一个积压队列长度
net.ipv4.tcp_timestamps=0
net.ipv4.ip_conntrack_max=65536
 
net.core.somaxconn = 32768 # 第二个积压队列长度
net.netfilter.nf_conntrack_max=2310720 #连接跟踪表的大小，建议根据内存计算该值CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (x / 32)，并满足nf_conntrack_max=4*nf_conntrack_buckets，默认262144  
net.bridge.bridge-nf-call-iptables=1 # 二层的网桥在转发包时也会被iptables的FORWARD规则所过滤，这样有时会出现L3层的iptables rules去过滤L2的帧的问题
net.bridge.bridge-nf-call-ip6tables=1 # 是否在ip6tables链中过滤IPv6包
 
vm.overcommit_memory=1 # 不检查物理内存是否够用 
vm.panic_on_oom=0 # 开启 OOM 
 
fs.may.detach_mounts=1
fs.file-max=52706963 # 文件描述符的最大值 
fs.nr_open=52706963 #设置最大微博号打开数 
fs.inotify.max_user_watches=524288 # 同一用户同时可以添加的watch数目，默认8192。

十、docker安装

　　1、所有节点安装docker-ce-19.03

1	`yum` `install` `-y docker-ce-19.03.*`

　　2、注：由于新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd

mkdir /etc/docker
 
cat > /etc/docker/daemon.json << EOF
{
    "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

　　3、所有节点设置开机自动启动docker

#启动docker
[root@k8s-master1 ~]# systemctl start docker
 
#设置开机启动
[root@k8s-master1 ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
 
#查看docker启动信息版本
[root@k8s-master1 ~]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)
 
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.15
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
.....等.......

　　注：查看docker信息的时候注意不要有警告信息

十一、安装Kubernetes(K8s)

　　1、查看k8s版本

1	`[root@k8s-master1 ~]# yum list kubeadm.x86_64 --showduplicates \| sort -r`

　　2、所有节点安装k8s组件

#直接安装kubeadm会安装最新的版本
yum install -y kubeadm
 
#对安装 kubeadm，kubectl,kubelet 指定1.20.6版本安装
yum install -y kubeadm-1.20.6-0.x86_64 kubectl-1.20.6-0.x86_64 kubelet-1.20.6-0.x86_64

　　3、所有节点配置。默认使用pause镜像使用gcr.io仓库，国内可能无法访问，所以这里配置Kubelet使用阿里云的pause镜像

cat > /etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"
EOF

　　4、设置Kubelet开机自动启动

[root@k8s-master1 ~]# systemctl daemon-reload
[root@k8s-master1 ~]# systemctl enable --now kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

十二、高可用组件安装(注：如果不是高可用集群，haproxy和keepalived无需安装)

　　1、所有k8s-master主节点(主节点:k8s-master1,k8s-master2,k8s-master3)通过yum安装HAProxy和KeepAlived

1	`yum` `install` `-y keepalived haproxy`

　　2、所有k8s-master主节点(主节点:k8s-master1,k8s-master2,k8s-master3)节点配置HAProxy(详细配置参考HAProxy文档，所有k8s主节点的HAProxy配置相同)

[root@k8s-master1 ~]# vim /etc/haproxy/haproxy.cfg
 
global
    maxconn     2000
    ulimit-n    16384
    log         127.0.0.1 local0 err
    stats timeout 30s
 
 
defaults
    mode                    http
    log                     global
    option                  httplog
    timeout connect         5000
    timeout client          50000
    timeout server          50000
    timeout http-request    15s
    timeout http-keep-alive 15s
 
 
frontend  monitor-in
    bind        *:33305
    mode        http
    option      httplog
    monitor-uri /monitor
 
 
 
frontend  k8s-master
    bind                0.0.0.0:16443
    bind                127.0.0.1:16443
    mode                tcp
    option              tcplog
    tcp-request         inspect-delay 5s
    default_backend     k8s-master
 
 
backend k8s-master
    mode        tcp
    option      tcplog
    option      tcp-check
    balance     roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 
    server      k8s-master1 192.168.3.123:6443 check
    server      k8s-master2 192.168.3.124:6443 check
    server      k8s-master3 192.168.3.125:6443 check

　　3、所有k8s主节点配置KeepAlived,配置文件不一样，注意区分每个k8s-master节点的IP地址和网卡(interface)

　　　　1）k8s-master1主节点配置文件(MASTER)

[root@k8s-master1 ~]# vim /etc/keepalived/keepalived.conf
 
! Configuration File for keepalived
 
global_defs {
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}
 
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
 
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.3.123
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.3.128 
    }
     
    track_script {
        chk_apiserver
    }
}

　　　　2）k8s-master2节点配置文件(BACKUP)

[root@k8s-master2 ~]# vim /etc/keepalived/keepalived.conf
 
! Configuration File for keepalived
 
global_defs {
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}
 
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
 
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    mcast_src_ip 192.168.3.124 
    virtual_router_id 51
    priority 99 
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.3.128
    }
     
    track_script {
        chk_apiserver
    }
}

　　　　3）k8s-master3节点配置文件(BACKUP)

[root@k8s-master3 ~]# vim /etc/keepalived/keepalived.conf
 
! Configuration File for keepalived
 
global_defs {
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}
 
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
 
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    mcast_src_ip 192.168.3.125 
    virtual_router_id 51
    priority 99 
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.3.128 
    }
     
    track_script {
        chk_apiserver
    }
}

注：

vrrp_script chk_apiserver {script "/etc/keepalived/check_apiserver.sh"} 这里是指定健康检测脚本路径
mcast_src_ip 192.168.3.123 这里 192.168.3.125 IP地址指的是本机IP地址
priority 100 这里指的是运行优先级(数字越大优先级越高)
virtual_ipaddress {192.168.3.128} 这里指的是vip地址(192.168.3.128)

　　4）所k8s-master节点配置KeepAlived健康检查脚本

[root@k8s-master1 ~]# vim /etc/keepalived/check_apiserver.sh
 
#!/bin/bash
#检测haproxy状态3次失败，关闭keepalived跳转到其它keepalived节点上面
 
err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done
 
if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi　

　　注：

　　　　1、通过KeepAlived虚拟出一个vip。vip会配置到k8s-master1节点上面(为虚拟主master)

　　　　2、健康检测会通过$(pgrep haproxy)检测haproxy端口。反带到haproxy配置文件设置bind 0.0.0.0:16443再反带到backend k8s-master1下面设置三个server节点上面

　　5）启动haproxy和keepalived

[root@k8s-master1 ~]# systemctl daemon-reload
[root@k8s-master1 ~]# systemctl enable --now haproxy
[root@k8s-master1 ~]# systemctl enable --now keepalived

　　检查haproxy和keepalived是否启动正常

#检查Haproxy
[root@k8s-master1 ~]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2022-08-23 20:01:33 CST; 59min ago
 Main PID: 5802 (haproxy-systemd)
    Tasks: 3
   Memory: 2.6M
   CGroup: /system.slice/haproxy.service
           ├─5802 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─5803 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─5804 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
 
#检查KeepAlived
[root@k8s-master1 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2022-08-23 19:14:00 CST; 1h 48min ago
  Process: 3647 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 3648 (keepalived)
    Tasks: 3
   Memory: 7.0M
   CGroup: /system.slice/keepalived.service
           ├─3648 /usr/sbin/keepalived -D
           ├─3649 /usr/sbin/keepalived -D
           └─3650 /usr/sbin/keepalived -D
 
8月 23 19:14:04 k8s-master1 Keepalived_vrrp[3650]: Sending gratuitous ARP on ens33 for 192.168.3.128
8月 23 19:14:04 k8s-master1 Keepalived_vrrp[3650]: Sending gratuitous ARP on ens33 for 192.168.3.128

　　检查vip IP：192.168.3.128是否正常，出现以下情况说明VIP IP地址正常

#用ping检查vip地址
[root@k8s-master1 ~]# ping 192.168.3.128
PING 192.168.3.128 (192.168.3.128) 56(84) bytes of data.
64 bytes from 192.168.3.128: icmp_seq=1 ttl=64 time=0.102 ms
64 bytes from 192.168.3.128: icmp_seq=2 ttl=64 time=0.079 ms
64 bytes from 192.168.3.128: icmp_seq=3 ttl=64 time=0.057 ms
64 bytes from 192.168.3.128: icmp_seq=4 ttl=64 time=0.298 ms
 
#通过telnet 检查16443端口vip IP地址
[root@k8s-master1 ~]# telnet 192.168.3.128 16443
Trying 192.168.3.128...
Connected to 192.168.3.128.
Escape character is '^]'.
Connection closed by foreign host.

　　注：

　　　　1、如果ping vip IP地址不通，且telnet不通，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙selinux , haproxy和keepalived的状态，监

　　　　听端口等。所有节点查看防火墙状态必须是disable和inactive; systemctl stauts firewalld所有节点查看selinux状态，必须为disable; getenforce

　　　　master 节点查看haproxy和keepalived状态：systemctl status keepalived/haproxy。master节点查看监听端口netstat -lntp是否正常开启

十三、集群初始化

　　1、生成 kubeadm 配置文件：

1	`kubeadm config print init-defaults > kubeadm-config.yaml`

　　2、对生成的 kubeadm-config.yaml配置文件修改，所有节点配置 kubeadm-config.yaml。所有节点修改后如下：

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.3.123
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - 192.168.3.128
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.3.128:16443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.6
networking:
  dnsDomain: cluster.local
  podSubnet: 172.100.0.0/12
  serviceSubnet: 10.96.0.0/12
scheduler: {}

注：

1、advertiseAddress: 192.168.3.123 绑定的是k8s-master1主节点IP地址
ertSANs

　　　- 192.168.3.128 绑定的是vip IP地址

controlPlaneEndpoint: 192.168.3.128:16443 这里指定的是VIP地址:haproxy端口
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers 指定阿里云下载镜像
kubernetesVersion: v1.20.6 指的是kubernetes安装运行的版本号
如果不是高可用集群，192.168.3.128:16443改为k8s-master1的地址，16443改为apiserver的端口，默认是6443，注意更改v1.20.6为自己服务器kubeadm的版本：kubeadm version

　　3、更新kubeadm文件转换成最新的(注：这里是把上面创建的kubeadm-config.yaml文件转换成新的new.yaml。如果kubeadm-config.yaml是最新的不需要执行下面的命令)

1	`kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml`

注：　

转换为new.yaml后在下载镜像的时候，需要将new.yaml文件复制到其它k8s(master)节点之后在下载镜像。new.yaml下载镜像如下

1	`kubeadm config images pull --config` `/root/new.yaml`

kubeadm-config.yaml配置文件位转换更新下载镜像命令如下：

1	`kubeadm config images pull --config` `/root/kubeadm-config.yaml`

　　4、所有节点设置开机自动启动kubeadm(注：如果上面设置kubelet开机自动启动，这里不需要在设置)

1	`systemctl` `enable` `--now kubelet`

　　注：如果启动失败无需管理，初始化成功以后即可启动

　　5、k8s-master1(master)节点初始化，初始化以后会在/etc/kubernetes目录下生产对应证书和配置文件，之后其它k8s(master)节点加入k8s-master1节点即可

1	`kubeadm init --config` `/root/kubeadm-config.yaml --upload-certs`

注：

初始化成功以后，会产生Token值，用于其它节点加入使用，因此要记录下初始化成功生成的token值(令牌值)。Token记录

Your Kubernetes control-plane has initialized successfully!
 
To start using your cluster, you need to run the following as a regular user:
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
Alternatively, if you are the root user, you can run:
 
  export KUBECONFIG=/etc/kubernetes/admin.conf
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
You can now join any number of the control-plane node running the following command on each as root:
 
  kubeadm join 192.168.3.128:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:e9b6b8bbe90be69f0176623ccb1ed77c06efeea1caa7e3da97d341776d6bc189 \
    --control-plane --certificate-key 49ecba55a13c8386a3d4446ed6a0c2187dcd8c63a9cc94214bc858cfce64966f
 
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
 
Then you can join any number of worker nodes by running the following on each as root:
 
kubeadm join 192.168.3.128:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:e9b6b8bbe90be69f0176623ccb1ed77c06efeea1caa7e3da97d341776d6bc189

　　6、如果初始化失败，重置后再次初始化，命令如下：

1	`kubeadm reset -f ; ipvsadm --clear` `;` `rm` `-fr ~/.kube`

　　7、k8s-master1节点配置环境变量，用于访问Kubernetes集群

cat <<EOF >> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc

　　注：/etc/kubernetes/admin.conf 文件用于集群通信

　　8、查看集群是否创建成功

[root@k8s-master1 ~]# kubectl get node
NAME    STATUS     ROLES                  AGE     VERSION
k8s-master1   NotReady   control-plane,master   7m11s   v1.20.6
 
 
[root@k8s-master1 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7m25s
 
#查看集群状态
 
[root@k8s-master1 ~]# kubectl get nodes
NAME    STATUS     ROLES                  AGE   VERSION
k8s-master1   NotReady   control-plane,master   11m   v1.20.6

　　10、采用初始化安装方式，所有的系统组件均以容器的方式运行并且在kube-system命令空间内，此时可以查看Pod状态

[root@k8s-master1 ~]# kubectl get pods -n kube-system -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
coredns-54d67798b7-6z2hl        0/1     Pending   0          7m44s   <none>          <none>   <none>           <none>
coredns-54d67798b7-gwdsn        0/1     Pending   0          7m44s   <none>          <none>   <none>           <none>
etcd-k8s-master1                      1/1     Running   0          8m2s    192.168.3.123   k8s-master1    <none>           <none>
kube-apiserver-k8s-master1            1/1     Running   0          8m2s    192.168.3.123   k8s-master1    <none>           <none>
kube-controller-manager-k8s-master1   1/1     Running   0          8m2s    192.168.3.123   k8s-master1    <none>           <none>
kube-proxy-psfdp                1/1     Running   0          7m44s   192.168.3.123   k8s-master1    <none>           <none>
kube-scheduler-k8s-master1            1/1     Running   0          8m2s    192.168.3.123   k8s-master1    <none>           <none>

　　注：系统组件都是在/etc/kubernetes/manifests/文件下面

　　11、k8s-master1初始化完成后对，把k8s-master2,k8s-master3, node-1,node-2节点加入k8s-mster1里面，需要使用的命令是初始化后记录的数据中一段命令。

首先把k8s-master2,k8s-master3加入k8s-master1里面命令如下(在k8s-mster1,k8s-master2上面运行)：

kubeadm join 192.168.3.128:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:e9b6b8bbe90be69f0176623ccb1ed77c06efeea1caa7e3da97d341776d6bc189 \
    --control-plane --certificate-key 49ecba55a13c8386a3d4446ed6a0c2187dcd8c63a9cc94214bc858cfce64966f

　　注：初始化生成的时候每条命令不一样，需以自己生成的为准进行操作

node-1,node-2节点加入k8s-master1里面，在node节点上面运行如下相关命令:

1 2	`kubeadm` `join` `192.168.3.128:16443 --token abcdef.0123456789abcdef \` `--discovery-token-ca-cert-hash` `sha256:e9b6b8bbe90be69f0176623ccb1ed77c06efeea1caa7e3da97d341776d6bc189 \`

　　注：node-1,node-2节点加入时没有--control-plane --certificate-key 这段参数值，只需按照上面把kubeadm jion ...等... --token ...等... --discovery-token-ca-cert-hash这段在node节点上输入运行

　查看节点插入后命令

[root@k8s-master1 ~]# kubectl get node
NAME          STATUS     ROLES                  AGE   VERSION
k8s-master1   NotReady   control-plane,master   20m35s   v1.20.6
k8s-master2   NotReady   control-plane,master   15m10s   v1.20.6
k8s-master3   NotReady   control-plane,master   10m20s   v1.20.6
node-1        NotReady   <none>                 7m11s   v1.20.6
node-2        NotReady   <none>                 7m45s   v1.20.6

　　12、Token过期后生成新的Token，在k8s-master1主节点上执行一些命令

1	`kubeadm token create --print-join-command`

　　k8s节点需要生成 --certificate-key

1	`kubeadm init phase upload-certs --upload-certs`

　　把生成新的token和--certificate-key替换一起保存的相关信息，重新执行即可

十四、Calico安装

　　1、以下步骤只在k8s-master1执行

cd /root; git clone https://github.com/dotbalo/k8s-ha-install.git
cd k8s-ha-install
git checkout manual-installation-v1.20.x

　　2、修改calico-etcd.yaml配置文件

修改etcd的节点

cd /root/k8s-ha-install/calico/
 
sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.3.123:2379,https://192.168.3.124:2379,https://192.168.3.125:2379"#g' calico-etcd.yaml

证书解析加密添加到配置文件中

ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml

把 etcd_key 放到 secret 里面，secret 会挂载到 calico 容器的 pod 里面，挂载的名称就是 ETCD_CA，这样 calico 就能找到证书，就可以连接到 etcd，就可以把 pod 信息存储到 etcd 里面

1	`sed` `-i` `'s#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g'` `calico-etcd.yaml`

修改 pod 网段

1 2	POD_SUBNET=``cat` `/etc/kubernetes/manifests/kube-controller-manager.yaml \|` `grep` `cluster-cidr= \|` `awk` `-F=` `'{print $NF}'`` `sed` `-i` `'s@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.0.0/16"@ value: '"${POD_SUBNET}"'@g'` `calico-etcd.yaml`

　　注：

　　　　这个步骤是把calico-etcd.yaml文件里面的CALICO_IPV4POOL_CIDR下的网段改成自己的Pod网段，也就是把192.168.x.x/16改成自己的集群网段。

　　　　并打开注释，所以更改的时候请确保这个步骤的这个网段没有被统一替换掉，如果被替换掉了，还请改回来　　

　　3、创建calico

cd /root/k8s-ha-install/calico/
 
kubectl apply -f calico-etcd.yaml

　　4、查看容器是否启动完成

[root@k8s-master1 dashboard]# kubectl get po -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
kube-system            calico-kube-controllers-5f6d4b864b-cwk6p     1/1     Running   1          18m
kube-system            calico-node-2j75v                            1/1     Running   2          18m
kube-system            calico-node-66vdj                            1/1     Running   0          18m
kube-system            calico-node-j9scc                            1/1     Running   1          18m
kube-system            calico-node-tkjdj                            1/1     Running   0          18m
kube-system            calico-node-v9v7x                            1/1     Running   0          18m
kube-system            coredns-54d67798b7-wtlfb                     1/1     Running   0          31m
kube-system            coredns-54d67798b7-zzn5g                     1/1     Running   0          31m
kube-system            etcd-k8s-master1                             1/1     Running   0          31m
kube-system            etcd-k8s-master2                             1/1     Running   0          28m
kube-system            etcd-k8s-master3                             1/1     Running   0          27m
kube-system            kube-apiserver-k8s-master1                   1/1     Running   0          31m
kube-system            kube-apiserver-k8s-master2                   1/1     Running   0          28m
kube-system            kube-apiserver-k8s-master3                   1/1     Running   0          27m
kube-system            kube-controller-manager-k8s-master1          1/1     Running   2          31m
kube-system            kube-controller-manager-k8s-master2          1/1     Running   0          28m
kube-system            kube-controller-manager-k8s-master3          1/1     Running   1          27m
kube-system            kube-proxy-6w5z6                             1/1     Running   0          31m
kube-system            kube-proxy-9dnxj                             1/1     Running   0          24m
kube-system            kube-proxy-dhdcj                             1/1     Running   0          25m
kube-system            kube-proxy-q28hc                             1/1     Running   0          28m
kube-system            kube-proxy-q67fk                             1/1     Running   0          27m
kube-system            kube-scheduler-k8s-master1                   1/1     Running   2          31m
kube-system            kube-scheduler-k8s-master2                   1/1     Running   0          28m
kube-system            kube-scheduler-k8s-master3                   1/1     Running   0          27m
kube-system            metrics-server-545b8b99c6-mgfjr              1/1     Running   0          14m
kubernetes-dashboard   dashboard-metrics-scraper-7645f69d8c-mxwq2   1/1     Running   0          13m
kubernetes-dashboard   kubernetes-dashboard-78cb679857-qvvwx        1/1     Running   0          13m

十五、Metrics Server部署

　　说明：在Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

　　1、将k8s-master1节点的front-proxy-ca.crt复制到所有Node节点

1 2	`[root@k8s-master1 calico]# scp /etc/kubernetes/pki/front-proxy-ca.crt node-1:/etc/kubernetes/pki/front-proxy-ca.crt` `[root@k8s-master1 calico]# scp /etc/kubernetes/pki/front-proxy-ca.crt node-2:/etc/kubernetes/pki/front-proxy-ca.crt`

　　2、关于配置文件修改（注：证书和镜像地址配置文件已经做好了修改可以直接使用。如果这两项在运行中失效再进行修改）

添加证书，不然可能导致获取不到度量指标

1	`- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem` `# change to front-proxy-ca.crt for kubeadm`

修改了镜像地址为阿里云

1	`image: registry.cn-beijing.aliyuncs.com/dotbalo/metrics-server:v0.4.1`

　　3、安装metrics server

cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm/
 
[root@k8s-master1 metrics-server-0.4.x-kubeadm]# kubectl  create -f comp.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

　　4、等待kube-system命令空间下的Pod全部启动后，查看状态

[root@k8s-master1 metrics-server-0.4.x-kubeadm]# kubectl top node
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master1   495m         24%    1184Mi          63%       
k8s-master2   434m         21%    1120Mi          59%       
k8s-master3   467m         23%    1220Mi          65%       
node-1        246m         12%    784Mi           41%       
node-2        212m         10%    712Mi           38% 

十六、Dashboard部署

　　查看配置文件修改镜像地址(下载的dashboard配置文件已经把镜像地址修改为了下方地址。这里可以不对dashboard配置文件做任何修改直接使用即可，如果镜像地址不能连接在进行修改)

image: registry.cn-beijing.aliyuncs.com/dotbalo/dashboard:v2.0.4
imagePullPolicy: Always
image: registry.cn-beijing.aliyuncs.com/dotbalo/metrics-scraper:v1.0.4

　　1、安装

cd /root/k8s-ha-install/dashboard/
 
[root@k8s-master1 dashboard]# kubectl  create -f .
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

　　2、更改dashboard的svc为NodePort：

1	`kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard`

　　注：将ClusterIP更改为NodePort（如果已经为NodePort忽略此步骤）

　　3、查看端口号

[root@k8s-master1 dashboard]# kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.97.80.76   <none>        443:31191/TCP   12m

　　注：端口31191是随机创建

　　4、查看容器是否启动完成

[root@k8s-master1 dashboard]# kubectl get po -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
kube-system            calico-kube-controllers-5f6d4b864b-cwk6p     1/1     Running   1          18m
kube-system            calico-node-2j75v                            1/1     Running   2          18m
kube-system            calico-node-66vdj                            1/1     Running   0          18m
kube-system            calico-node-j9scc                            1/1     Running   1          18m
kube-system            calico-node-tkjdj                            1/1     Running   0          18m
kube-system            calico-node-v9v7x                            1/1     Running   0          18m
kube-system            coredns-54d67798b7-wtlfb                     1/1     Running   0          31m
kube-system            coredns-54d67798b7-zzn5g                     1/1     Running   0          31m
kube-system            etcd-k8s-master1                             1/1     Running   0          31m
kube-system            etcd-k8s-master2                             1/1     Running   0          28m
kube-system            etcd-k8s-master3                             1/1     Running   0          27m
kube-system            kube-apiserver-k8s-master1                   1/1     Running   0          31m
kube-system            kube-apiserver-k8s-master2                   1/1     Running   0          28m
kube-system            kube-apiserver-k8s-master3                   1/1     Running   0          27m
kube-system            kube-controller-manager-k8s-master1          1/1     Running   2          31m
kube-system            kube-controller-manager-k8s-master2          1/1     Running   0          28m
kube-system            kube-controller-manager-k8s-master3          1/1     Running   1          27m
kube-system            kube-proxy-6w5z6                             1/1     Running   0          31m
kube-system            kube-proxy-9dnxj                             1/1     Running   0          24m
kube-system            kube-proxy-dhdcj                             1/1     Running   0          25m
kube-system            kube-proxy-q28hc                             1/1     Running   0          28m
kube-system            kube-proxy-q67fk                             1/1     Running   0          27m
kube-system            kube-scheduler-k8s-master1                   1/1     Running   2          31m
kube-system            kube-scheduler-k8s-master2                   1/1     Running   0          28m
kube-system            kube-scheduler-k8s-master3                   1/1     Running   0          27m
kube-system            metrics-server-545b8b99c6-mgfjr              1/1     Running   0          14m
kubernetes-dashboard   dashboard-metrics-scraper-7645f69d8c-mxwq2   1/1     Running   0          13m
kubernetes-dashboard   kubernetes-dashboard-78cb679857-qvvwx        1/1     Running   0          13m

　　注：

　　　　根据自己的实例端口号，通过任意安装了kube-proxy的宿主机或者VIP的IP+端口即可访问到dashboard：

　　　　访问Dashboard：https://192.168.3.123:31191（请更改31191为自己的端口），选择登录方式为令牌（即token方式）

　　　　也可以通过宿主机的ip访问：https://192.168.3.128:31191

　　5、查看token值，登录dashboard

[root@k8s-master1 dashboard]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
 
Name:         admin-user-token-c9ct2
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: bf5ba786-6bbe-437f-b35f-61279becd61d
 
Type:  kubernetes.io/service-account-token
 
Data
====
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkR1LU03eDRiYWNad2dMaWUyUkE4dmloNy1kRWdmMjdNREpmdVR2aG8wU0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOi
JrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWM5Y3QyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm
5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiZjViYTc4Ni02YmJlLTQzN2YtYjM1Zi02MTI3OWJlY2Q2MWQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.pRqv35ab291R0nEkbwKFF
WWfawJ6B-eXpF_3SgNI1-1DC9YwWV7PJ32jprwEKQlp7EoHj6Z39xRV5JnyodyeiEYPzLUupUX_B9irt8XoiaGUl2plMpgog-EXQr5qJiRE7D8dKyvjS84vvAHDjjEyq6Nl4-N4qxBv0OphVKwCZ7YMhxsi3uEqjwZ0Pzw-unkKl036dGtmy9XuX4vcOLBtO9Ruib_G0OCwV
VNqksDidJi4Twzj9LOXxil7G4R2X88Qm7AEy1SpOnonXum9SUYGpCAFmvpcYVYHrgpB0nl-zUAyD2u7ZiYxNVe79edy-yhLp0dokEIKad7H6ztkKiWcLQ
ca.crt:     1066 bytes

　　注：token:后面是查看到的token值，在登录dashboard的时候输入使用

访问dashboard输入图下

　登录界面为kube-system

十七、一些必须修改的配置更改(再k8s-master1节点执行)

　　1、再k8s-master1节点执行。将Kube-proxy改为ipvs模式，因为在初始化集群的时候注释了ipvs配置，所以需要自行修改一下命令如下：

1	`kubectl edit cm kube-proxy -n kube-system`

注：mode: ""修改为 mode: "ipvs"

　　2、更新 Kube-Proxy 的 Pod：(pod就会滚动更新)

1 2	`[root@k8s-master1 dashboard]`# kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system `daemonset.apps/kube-proxy` `patched`

　　3、查看 pod 滚动更新

[root@k8s-master1 dashboard]# kubectl get po -n kube-system -o wide
NAME                                       READY   STATUS    RESTARTS   AGE    IP                NODE          NOMINATED NODE   READINESS GATES
calico-kube-controllers-5f6d4b864b-cwk6p   1/1     Running   1          151m   192.168.3.123     k8s-master1   <none>           <none>
calico-node-2j75v                          1/1     Running   2          151m   192.168.3.123     k8s-master1   <none>           <none>
calico-node-66vdj                          1/1     Running   0          151m   192.168.3.126     node-1        <none>           <none>
calico-node-j9scc                          1/1     Running   5          151m   192.168.3.124     k8s-master2   <none>           <none>
calico-node-tkjdj                          1/1     Running   0          151m   192.168.3.127     node-2        <none>           <none>
calico-node-v9v7x                          1/1     Running   2          151m   192.168.3.125     k8s-master3   <none>           <none>
coredns-54d67798b7-wtlfb                   1/1     Running   0          164m   172.106.159.129   k8s-master1   <none>           <none>
coredns-54d67798b7-zzn5g                   1/1     Running   0          164m   172.106.159.130   k8s-master1   <none>           <none>
etcd-k8s-master1                           1/1     Running   3          164m   192.168.3.123     k8s-master1   <none>           <none>
etcd-k8s-master2                           1/1     Running   1          162m   192.168.3.124     k8s-master2   <none>           <none>
etcd-k8s-master3                           1/1     Running   0          161m   192.168.3.125     k8s-master3   <none>           <none>
kube-apiserver-k8s-master1                 1/1     Running   6          164m   192.168.3.123     k8s-master1   <none>           <none>
kube-apiserver-k8s-master2                 1/1     Running   2          162m   192.168.3.124     k8s-master2   <none>           <none>
kube-apiserver-k8s-master3                 1/1     Running   1          161m   192.168.3.125     k8s-master3   <none>           <none>
kube-controller-manager-k8s-master1        1/1     Running   3          164m   192.168.3.123     k8s-master1   <none>           <none>
kube-controller-manager-k8s-master2        1/1     Running   1          162m   192.168.3.124     k8s-master2   <none>           <none>
kube-controller-manager-k8s-master3        1/1     Running   1          161m   192.168.3.125     k8s-master3   <none>           <none>
kube-proxy-7wg2x                           1/1     Running   0          17s    192.168.3.123     k8s-master1   <none>           <none>
kube-proxy-8g6rt                           1/1     Running   0          36s    192.168.3.124     k8s-master2   <none>           <none>
kube-proxy-bw682                           1/1     Running   0          30s    192.168.3.126     node-1        <none>           <none>
kube-proxy-r4hgv                           1/1     Running   0          50s    192.168.3.127     node-2        <none>           <none>
kube-proxy-xdthg                           1/1     Running   0          41s    192.168.3.125     k8s-master3   <none>           <none>
kube-scheduler-k8s-master1                 1/1     Running   3          164m   192.168.3.123     k8s-master1   <none>           <none>
kube-scheduler-k8s-master2                 1/1     Running   1          162m   192.168.3.124     k8s-master2   <none>           <none>
kube-scheduler-k8s-master3                 1/1     Running   0          161m   192.168.3.125     k8s-master3   <none>           <none>
metrics-server-545b8b99c6-mgfjr            1/1     Running   2          148m   172.104.247.1     node-2        <none>           <none>

　　AGE时间显示的30s 50s 已经显示pod滚动更新完成

　　3、验证pod滚动更新为ipvs命令(node-2验证)

1	`curl 127.0.0.1:10249/proxyMode`

十八、注意事项

　　1、kubeadm安装的集群，证书有效期默认是一年。master节点的kube-apiserver、kube-scheduler、kube-controller-manager、etcd都是以容器运行的。可以通过kubectl get po -n kube-system查看

　　2、启动和二进制的区别

kubelet的配置文件在/etc/sysconfig/kubelet和/var/lib/kubelet/config.yaml，修改后需要重启kubelet进程
其他组件的配置文件在/etc/kubernetes/manifests目录下，比如kube-apiserver.yaml，该yaml文件更改后，kubelet会自动刷新配置，也就是会重启pod。不能再次创建该文件

　　3、Kubeadm安装后，master节点默认不允许部署pod，会占用资源，在学习过程中可以通过以下方式打开：

查看Taints产生的污点，命令如下

[root@k8s-master1 ~]# kubectl describe node -l node-role.kubernetes.io/master=  | grep Taints
Taints:             node-role.kubernetes.io/master:NoSchedule
Taints:             node-role.kubernetes.io/master:NoSchedule
Taints:             node-role.kubernetes.io/master:NoSchedule

　　注;通过命令查看到了有三个污点

删除Taints污点命令如下

[root@k8s-master1 ~]# kubectl  taint node  -l node-role.kubernetes.io/master node-role.kubernetes.io/master:NoSchedule-
node/k8s-master1 untainted
node/k8s-master2 untainted
node/k8s-master3 untainted

删除Taints污点,再查看是否删除完成

[root@k8s-master1 ~]# kubectl describe node -l node-role.kubernetes.io/master=  | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>

　　注：通过用查看污点命令，三个污点已经删除

十九、安装Kuboard

　　说明：Kuboard 是 Kubernetes 的一款图形化管理界面

　　1、安装

1	`kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml`

　　2、查看 Kuboard 运行状态

[root@k8s-master1 ~]# kubectl get pods -l k8s.kuboard.cn/name=kuboard -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
kuboard-74c645f5df-xj9fs   1/1     Running   0          100s

　　3、会生成一个svc，查看svc创建的端口

[root@k8s-master1 ~]# kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns         ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   29h
kuboard          NodePort    10.107.67.103   <none>        80:32567/TCP             8m54s
metrics-server   ClusterIP   10.97.217.111   <none>        443/TCP                  29h

　　注：kuboard 这行 80:32567/TCP。32567端口是访问kuboard到时候使用的

　　vip虚拟主机IP访问：192.168.3.128:32567

　　k8s-master1真是主机IP地址访问：192.168.3.123:32567

　　4、查看Token值作为登录密码，查看命令

1	`[root@k8s-master1 ~]# echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret \| grep ^kuboard-user \| awk '{print $1}') -o go-template='{{.data.token}}' \| base64 -d)`

　　5、kuboard官网安装连接地址：https://www.kuboard.cn/install/v3/install.html

posted @ 2022-08-30 18:12 点点滴滴的努力阅读(802) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· 二进制安装Kubernetes高可用集群

· Prometheus监控操作

· 使用kubeadm快速部署k8s高可用集群

· Kubeadm搭建K8s集群

· Kubeadm部署高可用K8S集群

阅读排行：
· 无需6万激活码！GitHub神秘组织3小时极速复刻Manus，手把手教你使用OpenManus搭建本
· C#/.NET/.NET Core优秀项目和框架2025年2月简报
· 什么是nginx的强缓存和协商缓存
· 一文读懂知识蒸馏
· Manus爆火，是硬核还是营销？

点点滴滴的努力

Kubeadm安装Kubernetes高可用集群

十三、集群初始化

十五、Metrics Server部署

十六、Dashboard部署

十七、一些必须修改的配置更改(再k8s-master1节点执行)

十八、注意事项

十九、安装Kuboard

搜索

常用链接

随笔分类

阅读排行榜

评论排行榜

推荐排行榜

最新评论