k8s-kubeadm高可用安装部署

Kubernetes是谷歌以Borg为前身，基于谷歌15年生产环境经验的基础上开源的一个项目
Kubernetes致力于提供跨主机集群的自动部署、扩展、高可用以及运行应用程序容器的平台

Kubernetes Master节点：

• kube-APIServer: 集群的控制中枢，各个模块之间信息交互都需要经过 kube-apiserver，同时它也是集群管理、资源配置、整个集群安全机制的入口。
• Controller-Manager: 集群的状态管理器，保证Pod或其他资源达到期望值，也是需要和 apiserver 进行通信，在需要的时候创建、更新、删除它所管理的资源。
• Scheduler: 集群的调度中心，它会根据指定的一系列条件，选择一个或一批最佳的节点，然后部署Pod。
• Etcd：键值数据库，保存一些集群的信息，一般生产环境中建议部署三个以上节点(奇数个)

基本环境配置

kubeadm 安装方式自 1.14 版本以后，安装方法几乎没有任何变化，本文可以尝试安装最新的k8s集群

[root@c71 ~]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@c71 ~]# uname -a
Linux c71 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

k8s官网
k8s官网kubeadm高可用安装
kubernetes部署安装包4xnf

集群规划

主机规划

主机名	IP地址	说明
k8s-master01-03	192.168.10.10/20/30	master节点*3
k8s-master-lb	192.168.10.100	keepalived虚拟VIP
k8s-node01-03	192.168.10.40/50/60	worker节点*3

网段规划

配置信息	备注
系统版本	centos7.9
docker版本	20.10.x
pod网段	172.16.0.0/12
service网段	10.96.0.0/16

宿主机网段、k8s service网段、pod网段不能重复

所有机器修改主机名与hosts文件

[root@c71 ~]# hostnamectl set-hostname k8s-master01
[root@c71 ~]# vim /etc/hosts
[root@c71 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.10 k8s-master01
192.168.10.20 k8s-master02
192.168.10.30 k8s-master03
192.168.10.100 k8s-master-lb
192.168.10.40 k8s-node01
192.168.10.50 k8s-node02
192.168.10.60 k8s-node03

配置yum源

wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum -y install yum-utils device-mapper-persistent-data lvm2 git telnet net-tools vim psmisc jq wget ntpdate
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

所有节点关闭 firewalld、dnsmasq、selinux(CentOS7需要关闭 NetworkManager，CentOS8不需要)

systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

所有节点关闭swap分区，fstab注释swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

所有节点同步时间
安装ntpdate

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm

所有节点同步时间。时间同步配置如下：

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

所有节点配置limit：

ulimit -SHn 65535
vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited　

Master01节点免密钥登录其他节点，安装过程中生成配置文件和证书均在Master01上操作，集群管理也在Master01上操作，阿里云或者AWS上需要单独一台kubectl服务器。
配置方法：略

下载配置文件

cd /root/ ; git clone git@github.com:Chuyio/kubernetes.git

升级内核

所有节点升级系统并重启

yum update -y --exclude=kernel* && reboot #CentOS7需要升级，CentOS8可以按需升级系统

CentOS7 需要升级内核至4.18+，本次升级的版本为4.19
在master01节点下载内核

cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

下载后从master01节点传到其他节点
可以使用以下脚本进行快速传至其他节点，或者使用ansible

[root@k8s-master01 ~]# vim scp_file.sh
#!/bin/bash
PC_NODE=$1
FILE_PATH=$2

MASTER_FILE() {
for MASTER in {k8s-master02,k8s-master03}
do
    scp ${FILE_PATH} $MASTER:${FILE_PATH}
done
}
NODE_FILE() {
for NODE in {k8s-node01,k8s-node02,k8s-node03}
do
    scp ${FILE_PATH} $NODE:${FILE_PATH}
done
}
case $PC_NODE in
master)
    MASTER_FILE
;;
node)
    NODE_FILE
;;
all)
    MASTER_FILE
    NODE_FILE
;;
esac
[root@k8s-master01 ~]# chmod +x scp_file.sh 
[root@k8s-master01 ~]# ln -s /root/scp_file.sh /usr/local/bin/

使用方法

[root@k8s-master01 ~]# for i in $(ls /root/kernel-ml*); do scp_file.sh all $i; done
kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm                                                  100%   46MB 134.2MB/s   00:00    
kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm                                                  100%   46MB 122.9MB/s   00:00    
kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm                                                  100%   46MB 131.3MB/s   00:00    
kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm                                                  100%   46MB 139.4MB/s   00:00    
kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm                                                  100%   46MB 133.8MB/s   00:00    
kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm                                            100%   12MB  97.0MB/s   00:00    
kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm                                            100%   12MB 103.6MB/s   00:00    
kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm                                            100%   12MB  94.8MB/s   00:00    
kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm                                            100%   12MB  81.5MB/s   00:00    
kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm                                            100%   12MB  88.4MB/s   00:00

所有节点安装内核

cd /root && yum localinstall -y kernel-ml*

所有节点更改内核启动顺序

grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

检查默认内核是不是4.19

[root@k8s-master01 ~]# grubby --default-kernel
/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64

所有节点重启，然后检查内核是不是4.19

[root@k8s-master01 ~]# uname -a
Linux k8s-master01 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

所有节点安装ipvsadm

yum install ipvsadm ipset sysstat conntrack libseccomp -y

所有节点配置 ipvs 模块，在内核4.19+版本 nf_conntrack_ipv4 已经改为 nf_conntrack， 4.18以下使用 nf_conntrack_ipv4 即可

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
vim /etc/modules-load.d/ipvs.conf 
	# 加入以下内容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

然后执行

systemctl enable --now systemd-modules-load.service

检查是否加载

[root@k8s-master01 ~]# lsmod | grep -e ip_vs -e nf_conntrack
ip_vs_ftp              16384  0 
nf_nat                 32768  1 ip_vs_ftp
ip_vs_sed              16384  0 
ip_vs_nq               16384  0 
ip_vs_fo               16384  0 
ip_vs_dh               16384  0 
ip_vs_lblcr            16384  0 
ip_vs_lblc             16384  0 
ip_vs_wlc              16384  0 
ip_vs_lc               16384  0 
ip_vs_sh               16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  0 
ip_vs                 151552  24 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack          143360  2 nf_nat,ip_vs
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  4 nf_conntrack,nf_nat,xfs,ip_vs

开启一些k8s集群中必须的内核参数，所有节点配置k8s内核

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
net.ipv4.conf.all.route_localnet = 1

vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system

所有节点配置完内核后，重启服务器，保证重启后内核依旧加载

reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

基本组件安装

如果安装得版本低于1.24，选择 Docker 和 Containerd 均可，高于1.24选择 Containerd 作为 Runtime。

配置kubernetes yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

Containerd作为Runtime

版本大于等于1.24

所有节点安装 docker-ce-20.10*

[root@k8s-master02 ~]# yum -y install docker-ce-20.10.* docker-ce-cli-20.10.*

可以无需启动docker，只需要配置和启动Containerd即可
首先配置Containerd所需得模块(所有节点)

[root@k8s-master01 ~]# cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter

所有节点加载模块

[root@k8s-master01 ~]# modprobe -- overlay
[root@k8s-master01 ~]# modprobe -- br_netfilter

所有节点配置 Containerd 所需得内核

[root@k8s-master01 ~]# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1

所有节点加载内核

[root@k8s-master01 ~]# sysctl --system

所有节点配置Containerd得配置文件

[root@k8s-master01 ~]# cd /etc/containerd/
[root@k8s-master01 containerd]# containerd config default | tee /etc/containerd/config.toml #重新生成一个
[root@k8s-master01 containerd]# vim config.toml
修改以下配置，将false改为true
125             SystemdCgroup = true
所有节点将sandbox_image的Pause镜像改成符合自己版本的地址registry.cn-hangzhou.aliyuncs.com/google_containers/pause
 61     sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

所有节点启动Containerd，并配置开机自启动

[root@k8s-master01 containerd]# systemctl daemon-reload
[root@k8s-master01 containerd]# systemctl enable --now containerd

所有节点配置crictl客户端连接的运行时位置

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

docker作为Runtime

所有节点安装docker-ce 20.10

yum install docker-ce-20.10.* docker-ce-cli-20.10.* -y

由于新版Kubelet建议使用systemd，所以把Docker的CgroupDriver也改成systemd

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

所有节点设置开机自启动Docker

systemctl daemon-reload && systemctl enable --now docker

安装Kubernetes组件

在master01上查看最新得版本是多少

[root@k8s-master01 ~]# yum list kubeadm.x86_64 --showduplicates | sort -r
···
看到目前最新得是1.24.2
kubeadm.x86_64                       1.24.2-0                         kubernetes
···

所有节点安装1.24.2版本 kubeadm、kubelet和kubectl

yum -y install kubeadm-1.24.2* kubelet-1.24.2* kubectl-1.24.2*

查看版本

[root@k8s-master01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:20:54Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

如果选择得是Containerd作为runtime，需要更改kubelet得配置使用Containerd作为Runtime
如果是用docker作为runtime，不要执行下面得命令

cat >/etc/sysconfig/kubelet<<EOF
KUBELET_KUBEADM_ARGS="--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF

所有节点设置kubelet开机自启动(由于还未初始化，没有kubelet得配置文件，kubelet无法启动，不用进行处理)

systemctl daemon-reload
systemctl enable --now kubelet

高可用组件安装

如果不是高可用集群，haproxy和keepalived无需安装

公有云要用云自带得负载均衡，如阿里云得SLB，腾讯云得ELB，云上环境大部分都不支持keepalived，阿里云kubectl控制端不能放在master节点，使用腾讯云可以解决阿里云SLB有回环，也就是SLB代理得服务器不能反向访问SLB，腾讯云修复了这个问题
所有 master 节点通过yum安装haproxy和keepalived

yum -y install haproxy keepalived

master节点配置haproxy

(详细配置参考haproxy文档)所有master节点得haproxy配置一样

vim /etc/haproxy/haproxy.cfg
# 注意更改IP地址
global
    log         127.0.0.1 local0 err
    maxconn     2000
    ulimit-n    16384
    daemon
    stats timeout 30s

defaults
    mode                    http
    log                     global
    option                  httplog
    timeout http-request    15s
    timeout connect         5000
    timeout client          50000
    timeout server          50000
    timeout http-keep-alive 15s


frontend monitor-in
    bind *:33305
    mode http
    option httplog
    monitor-uri /monitor

frontend k8s-master
    bind 0.0.0.0:16443
    bind 127.0.0.1:16443
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    default_backend k8s-master


backend k8s-master
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server k8s-master01 192.168.10.10:6443 check
    server k8s-master02 192.168.10.20:6443 check
    server k8s-master03 192.168.10.30:6443 check

启动haproxy

systemctl restart haproxy
netstat -anpt | grep haproxy
tcp        0      0 0.0.0.0:33305           0.0.0.0:*               LISTEN      4537/haproxy        
tcp        0      0 127.0.0.1:16443         0.0.0.0:*               LISTEN      4537/haproxy        
tcp        0      0 0.0.0.0:16443           0.0.0.0:*               LISTEN      4537/haproxy

master节点配置keepalived

配置都不一样，注意区分
master01

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}

vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2
   rise 1
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32	#本机网卡名
    mcase_src_ip 192.168.10.10	#本机IP地址
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.100	#VIP地址
    }
    track_script {
      chk_apiserver
    }
}

master02

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}

vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2
   rise 1
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcase_src_ip 192.168.10.20
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.100
    }
    track_script {
      chk_apiserver
    }
}

master03

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
   script_user root
   enable_script_security
}

vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2
   rise 1
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcase_src_ip 192.168.10.30
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.100
    }
    track_script {
      chk_apiserver
    }
}

所有master节点配置keepalived健康检查文件

[root@k8s-master01 ~]# cat /etc/keepalived/check_apiserver.sh 
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

启动keepalived

chmod +x /etc/keepalived/check_apiserver.sh
systemctl daemon-reload
systemctl enable --now keepalived

所有节点测试VIP

# ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.014 ms

集群初始化

官方初始化文件
master01节点创建 kubeadm-config.yaml 配置文件

如果不是高可用集群，192.168.10.100改为master01得地址，16443改为apiserver得端口，默认6443，注意更改kubernetesVersion值和自己得kubeadm得版本一致

[root@k8s-master01 ~]# vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.10
  bindPort: 6443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - 192.168.10.100  #改为VIP或云上LB负载均衡地址
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.10.10:16443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.24.2 # 更改此处的版本号和kubeadm version一致
networking:
  dnsDomain: cluster.local
  podSubnet: 172.16.0.0/12
  serviceSubnet: 10.96.0.0/16
scheduler: {}

更新kubeadm文件

[root@k8s-master01 ~]# kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml
W0626 21:23:09.971283   15405 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!

把 new.yaml 文件复制到其他master节点

[root@k8s-master01 ~]# scp_file.sh master new.yaml

kubeadm安装得k8s集群，组件是采用pod得形式去管理得

所有master节点提前下载组件pod镜像，可以节省初始化时间

其他master节点不需要更改任何配置包括IP地址

# kubeadm config images pull --config /root/new.yaml
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.24.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.24.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.24.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.24.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6

所有节点设置kubelet开机启动

启动失败无需处理，初始化完成后可正常启动
# systemctl enable --now kubelet  

初始化后会在 /etc/kubernetes 目录下生成对应得证书和配置文件，之后其他得master节点加入master01即可
master01节点初始化

[root@k8s-master01 ~]# kubeadm init --config /root/new.yaml  --upload-certs
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.10 192.168.10.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
W0626 21:37:49.572961   17090 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
W0626 21:37:49.701434   17090 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
W0626 21:37:49.811791   17090 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
W0626 21:37:49.930033   17090 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.524906 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
c3efed12fa4fa6f7c3b8c50f438951996143da6a4f1f67bf9838aa37123b374e
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7t2weq.bjbawausm0jaxury
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
W0626 21:38:12.849166   17090 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

# 初始化成功后会产生Token值，用于其他master节点加入时使用
  kubeadm join 192.168.10.10:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:6480cf4c4d91eb5ad978ba02dfefd27ec8688321f872d3a21d030eecdef46c34 \
	--control-plane --certificate-key c3efed12fa4fa6f7c3b8c50f438951996143da6a4f1f67bf9838aa37123b374e

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

# 初始化成功后会产生Token值，用于其他node节点加入时使用
kubeadm join 192.168.10.10:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:6480cf4c4d91eb5ad978ba02dfefd27ec8688321f872d3a21d030eecdef46c34 

初始化失败，需要重置后再次初始化

# kubeadm reset -f ; ipvsadm --clear ; rm -rf ~/.kube

测试kubectl命令，报错提示 localhost:8080 连不上，是因为kubectl连接不到k8s集群

[root@k8s-master01 ~]# kubectl get node
The connection to the server localhost:8080 was refused - did you specify the right host or port?

因为kubectl连接k8s集群需要有证书得文件
证书文件在 /etc/kubernetes/admin.conf
所以在执行kubectl命令时要指定 /etc/kubernetes/admin.conf 才能去连接到k8s集群
指定配置文件得方式，初始化集群后给出了两种,执行以下任意一种即可

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

执行以下命令

[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
执行后可以看到kubectl命令可以正常连接到k8s集群
[root@k8s-master01 ~]# kubectl get node
NAME           STATUS     ROLES           AGE   VERSION
k8s-master01   NotReady   control-plane   12m   v1.24.2

Master加入集群

master节点加入集群

kubeadm join 192.168.10.10:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:6480cf4c4d91eb5ad978ba02dfefd27ec8688321f872d3a21d030eecdef46c34 \
	--control-plane --certificate-key c3efed12fa4fa6f7c3b8c50f438951996143da6a4f1f67bf9838aa37123b374e

Token过期

新建得集群token有效期是24小时
查看secret，其中 bootstrap-token 是存放token得位置

[root@k8s-master01 ~]# kubectl get secret -n kube-system
NAME                     TYPE                            DATA   AGE
bootstrap-token-7t2weq   bootstrap.kubernetes.io/token   6      25m
bootstrap-token-lslz41   bootstrap.kubernetes.io/token   4      25m
kubeadm-certs            Opaque                          8      25m

在 new.yaml 中也能看到 7t2weq 与上面查出得 bootstrap-token-7t2weq 相对应

[root@k8s-master01 ~]# vim new.yaml 

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s

查看一下该 bootstrap-token-7t2weq 详情

[root@k8s-master01 ~]# kubectl get secret -n kube-system bootstrap-token-7t2weq -oyaml
apiVersion: v1
data:
  auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4=
  expiration: MjAyMi0wNi0yN1QxMzozODoxMlo=
  token-id: N3Qyd2Vx
  token-secret: YmpiYXdhdXNtMGpheHVyeQ==
  usage-bootstrap-authentication: dHJ1ZQ==
  usage-bootstrap-signing: dHJ1ZQ==
kind: Secret
metadata:
  creationTimestamp: "2022-06-26T13:38:12Z"
  name: bootstrap-token-7t2weq
  namespace: kube-system
  resourceVersion: "277"
  uid: 90682502-eb71-45ec-bf42-a2529df16451
type: bootstrap.kubernetes.io/token

可以看到 expiration(过期时间): MjAyMi0wNi0yN1QxMzozODoxMlo= 是加密过得
解密后便可看到真实得过期时间

[root@k8s-master01 ~]# echo "MjAyMi0wNi0yN1QxMzozODoxMlo=" | base64 -d
2022-06-27T13:38:12Z    #这个时间并不是当前时区，需要加8个小时
[root@k8s-master01 ~]# date
Sun Jun 26 22:10:09 CST 2022   # 与当前时间对比距离过期还有22个小时左右

Token过期后生成新得token
工作节点加入集群得token

kubeadm tolen create --print-join-command

master需要生成 --certificate-key

kubeadm init phase upload-certs --upload-certs

Node节点加入集群

node节点执行

kubeadm join 192.168.10.10:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:6480cf4c4d91eb5ad978ba02dfefd27ec8688321f872d3a21d030eecdef46c34 

都加入集群后master节点可以看到

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS     ROLES           AGE   VERSION
k8s-master01   NotReady   control-plane   20m   v1.24.2
k8s-master02   NotReady   control-plane   27s   v1.24.2
k8s-master03   NotReady   control-plane   25s   v1.24.2
k8s-node01     NotReady   <none>          18s   v1.24.2
k8s-node02     NotReady   <none>          15s   v1.24.2
k8s-node03     NotReady   <none>          4s    v1.24.2

Calico组件安装

关于网络插件得对比可以参考
K8s网络插件Flannel,Calico
Kubernetes网络插件对比分析(Flannel,Calico,Weave)

• flannel：支持网络管理，但是不支持网络策略
• calico：支持网络管理和网络策略
  master01上操作

[root@k8s-master01 ~]# cd ~/kubernetes/manual-installation-v1.24.x/ && cd calico
编辑calico.yaml文件
[root@k8s-master01 calico]# vim calico.yaml
4578               value: "POD_CIDR"
修改为POD得网段
4578               value: "172.16.0.0/12"

安装calico

[root@k8s-master01 calico]# kubectl apply -f calico.yaml

查看calico是否安装完成

[root@k8s-master01 ~]# kubectl get pod -n kube-system

安装完成后查看状态是否为Ready状态

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   65m   v1.24.2
k8s-master02   Ready    control-plane   45m   v1.24.2
k8s-master03   Ready    control-plane   45m   v1.24.2
k8s-node01     Ready    <none>          44m   v1.24.2
k8s-node02     Ready    <none>          44m   v1.24.2
k8s-node03     Ready    <none>          44m   v1.24.2

Metrics部署

在新版的kubernetes中系统的采集均使用metrics-server，可以通过Metrics采集节点和pod的内存、磁盘、cpu和网络的使用情况

把master01节点的front-proxy-ca.crt复制到所有node节点

[root@k8s-master01 ~]# scp_file.sh node /etc/kubernetes/pki/front-proxy-ca.crt front-proxy-ca.crt                           100% 1115     1.6MB/s   00:00    
front-proxy-ca.crt                           100% 1115     1.3MB/s   00:00    
front-proxy-ca.crt                           100% 1115     1.8MB/s   00:00 

安装metrics server

[root@k8s-master01 ~]# cd ~/kubernetes/manual-installation-v1.24.x/kubeadm-metrics-server/
[root@k8s-master01 kubeadm-metrics-server]# kubectl create -f comp.yaml

查看状态

[root@k8s-master01 ~]# kubectl get pod -n kube-system -l k8s-app=metrics-serverNAME                              READY   STATUS    RESTARTS   AGE
metrics-server-6c48d8f6d6-lpk9p   1/1     Running   0          7m7s

等待 metrics server 安装完成,可以查看资源使用情况

[root@k8s-master01 ~]# kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   132m         2%     1308Mi          34%       
k8s-master02   118m         2%     1367Mi          35%       
k8s-master03   100m         2%     1007Mi          26%       
k8s-node01     39m          0%     516Mi           13%       
k8s-node02     45m          1%     547Mi           14%       
k8s-node03     38m          0%     464Mi           12%

Dashboard部署

Dashboard 用于展示集群中的各类资源，同时也可以通过Dashboard实时查看pod的日志和在容器中执行一些命令等

安装指定版本的Dashboard

[root@k8s-master01 ~]# cd kubernetes/manual-installation-v1.24.x/dashboard/     
[root@k8s-master01 dashboard]# kubectl create -f .

查看状态

[root@k8s-master01 ~]# kubectl get pod -n kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-5b997cd56f-k2sfg   1/1     Running   0          6m23s
kubernetes-dashboard-5db64fd789-qh5fk        1/1     Running   0          6m23

查看暴露的端口号，暴露的端口号会不一样，请自行查询，我这里为 :31015

[root@k8s-master01 ~]# kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.96.122.64    <none>        8000/TCP        9m43s
kubernetes-dashboard        NodePort    10.96.115.119   <none>        443:31015/TCP   9m43s

若上面查询的TYPE栏显示的kubernetes-dashboard不是为NodePort则需要将svc更改为NodePort模式

Kubectl edit svc kubernetes-dashboard –n kubernetes-dashboard

本文已在配置文件改为NodePort，所以不用再更改

[root@k8s-master01 ~]# vim ~/kubernetes/manual-installation-v1.24.x/dashboard/dashboard.yaml 

查看type是否为nodeport，查看端口号

kubectl get svc kubernetes-dashboard -n kubernetes-dashboard

安装最新版本的Dashboard

官方Github地址

登录Dashboard

在谷歌浏览器Chrome启动快捷方式中加入启动参数，用于解决无法访问Dashboard的问题

--test-type --ignore-certificate-errors

登录dashboard

https+主机IP+暴露的端口号

创建管理员用户

cat > admin.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user

  namespace: kube-system
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding 
metadata: 
  name: admin-user
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:

- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

EOF

执行yaml文件

kubectl apply -f admin.yaml -n kube-system

serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created

已经创建用户的话，查询token

[root@k8s-master01 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

本文安装的是k8s1.24.版本，测试发现之前的查询token方式不能使用，上面查询及配置方法适用于1.24.之前的版本
k8s1.24.*创建admin-user用户，及查询token

[root@k8s-master01 ~]# cd /root/kubernetes/manual-installation-v1.23.x/dashboard
[root@k8s-master01 dashboard]# kubectl delete -f dashboard-user.yaml   # 删除之前创建的用户
[root@k8s-master01 ~]# wget https://raw.githubusercontent.com/cby-chen/Kubernetes/main/yaml/dashboard-user.yaml
[root@k8s-master01 ~]# kubectl apply -f dashboard-user.yaml
[root@k8s-master01 ~]# kubectl -n kubernetes-dashboard create token admin-user  #创建token
eyJhbGciOiJSUzI1NiIsImtpZCI6IkUzQWlSeW95X20xU3pOOTFWVTJoRC0tRHpIZ1lNci1UQlFwNXZ3cHFjUTgifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU2Mzg1MDM0LCJpYXQiOjE2NTYzODE0MzQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiMTdjZDdjODctZmYxYy00OGEzLWJiMDAtY2ZmMGE1YjcxMTU0In19LCJuYmYiOjE2NTYzODE0MzQsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.o4G0K4c3R0cHc0X-X1mpL6KYd-X0YfnuNDX1l8zVXfZTyTEjir6fnL2HBDRq1uBSV2lcbqiaZQCLa1EjVw4Jmtm6V6cAMNJzI9Mbzyf2A9z2ipvINj8WP_9z8DER27YN7SFctlVWV1Z5CqVGPfSXGwRflMxMuqiiN1p8rMzP5rlS5sQ4rm5BTMpFjRAZzjsgg6yuXELcRElX4UPpatjOdFkCKoor5MOINn3hNUw8JsmV2oGh8TACj25AjvNtELDV4-969nNjg7SHJCtElIuGIE3Bj-QZETtRVo2FWt229j44M9iqj82gRMKJOE33QdDtm6dgmtYnvCKvi97WloeIMQ

由于集群重启过，端口会发生变化，再次查询dashboard暴露的端口

[root@k8s-master01 ~]# kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.96.79.2      <none>        8000/TCP        22h
kubernetes-dashboard        NodePort    10.96.202.195   <none>        443:30059/TCP   22h

输入创建的token


没有安装metrics server是没有监控数据的

关键性配置

Kubernetes Sevice工作模式

Service的工作模式有三种

• ipvs
• iptables
• userspace

目前由于userspace太老已经启用，目前主流的为ipvs与iptables
使用 IPVS 在大规模集群下有一定的性能优势，但是也是需要一定条件才能支持的，如果没有满足的话就会自动降级为使用iptables模式，内核升级为 4.1 版本才完全支持。
在 Kubernetes 集群中，每个 Node 运行一个 kube-proxy 进程。kube-proxy 负责为 Service 实现了一种 VIP(虚拟IP)的形式，而不是 ExternalName 的形式。
在 Kubernetes v1.0 版本，代理完全在 userspace。在 Kubernetes v1.1 版本，新了 iptables 代理，但并不是默认的运行模式。从 Kubernetes v1.2 起，默认就是iptables代理。在 Kubernetes v1.8.0-beta.0 中，添加了 ipvs 代理。在 Kubernetes 1.14 版本开始默认使用 ipvs 代理。

Kube-proxy 是服务路由的构建块，它依赖于经过强化攻击的 iptables 来实现支持核心的服务类型，如 ClusterIP 和 NodePort。 但是，iptables 难以扩展到成千上万的服务，因为它纯粹是为防火墙而设计的，并且基于内核规则列表。

尽管 Kubernetes 在版本v1.6中已经支持5000个节点，但使用 iptables 的 kube-proxy 实际上是将集群扩展到5000个节点的瓶颈。 一个例子是，在5000节点集群中使用 NodePort 服务，如果我们有2000个服务并且每个服务有10个 pod，这将在每个工作节点上至少产生20000个 iptable 记录，这可能使内核非常繁忙。

另一方面，使用基于 IPVS 的集群内服务负载均衡可以为这种情况提供很多帮助。 IPVS 专门用于负载均衡，并使用更高效的数据结构（哈希表），允许几乎无限的规模扩张。

内核升级

本文前期已经完成升级内核步骤，如果已经内核版本已经大于4.1的可以忽略这一步骤，因为此内核版本可以完美的支持IPVS
如果没升级内核版本强行把Sevice模式由 iptables 改成 ipvs，可能会出现以下问题

• ipvs 指向的后端节点不对，如果重新 svc 删除重建又是对的
• 当副本数量变动时，ipvs 不能立马感知到
• 如果直接用域名访问 svc 时，会出现解析不了的情况

将kube-proxy改为ipvs模式

service底层实现目前主要由两个网络模式组成：iptables与IPVS。他们都是由kube-proxy维护
查看kube-proxy的模式，可以看到默认为iptables

[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
iptables

kubeadm的安装方式的配置文件是由 ConfigMap 去管理的
所以kube-proxy的配置文件是由 ConfigMap 去管理的
修改kube-proxy的配置文件，改为ipvs模式

[root@k8s-master01 ~]# kubectl edit cm kube-proxy -n kube-system

查找/mode 配置默认为空，空的默认就是iptables模式，改为ipvs

更改后滚动更新一下

[root@k8s-master01 ~]# kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system

更新后pod会进行重启

再次查看kube-proxy的模式

[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
ipvs

注意事项

kubeadm安装的集群，证书有效期默认是一年。master节点的kube-apiserver、kube-scheduler、kube-controller-manager、etcd都是以容器的方式运行的，可以通过kubectl get pod -n kube-system查看
启动和二进制不同的是
kubelet的配置文件在/etc/sysconfig/kubelet和/var/lib/kubelet/config.yaml。修改后需要重启kubelet进程
其他组件的配置文件在/etc/kubernetes/manifests目录下，比如kube-apiserver.yaml该yaml文件修改后，kubelet会自动刷新配置，也就是重启pod，不能再次创建该文件
kube-proxy的配置在kube-system命名空间下的configmap中，可以通过 kubectl edit cm kube-proxy -n kube-system 进行更改，更改后，可以通过patch重启kube-proxy

kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system