k8s1.20二进制包安装

集群环境准备

主机规划

主机IP地址	主机名	主机角色	主机配置	软件列表
10.58.32.31	k8s-master01	master,LB	2C4G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker，haproxy、keepalived
10.58.32.32	k8s-master02	master，LB	2C4G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker，haproxy、keepalived
10.58.32.33	k8s-master03	master，LB	2C4G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker，haproxy、keepalived
10.58.32.34	k8s-node01	worker	1C2G	kubelet、kube-proxy、docker
192.168.10.100	/	VIP(虚拟IP)	/

软件版本

软件名称	版本	备注
Rocky8	kernel版本：5.4
kubernetes	v1.20.15
etcd	v3.5.2	最新版本
calico	v3.4.13
coredns	v1.8.4
docker	1.19
haproxy	5.18	YUM源默认
keepalived	3.5	YUM源默认

网络分配

网络名称	网段	备注
Node网络	10.58.32.30~60/24
Service网络	10.96.0.0/16
Pod网络	172.16.0.0/12

参考：25.基于二进制包安装kubernetes v1.21 --集群部署(二) - 掘金 (juejin.cn)

安装Rocky 8

Vmware 安装

Node网络用的是NAT模式，然后又加了一个网卡，用于仅主机模式

系统初始化

升级系统

yum update -y --exclude=kernel*

更新内核

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install  -y https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
yum --enablerepo=elrepo-kernel install kernel-lt -y

查看目前可用内核

awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg

使用序号为0的内核，序号0是前面查出来的可用内核编号

grub2-set-default 0

生成 grub 配置文件并重启

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

修改网络IP

保证每台机器的网络UUID不同, 去掉UUID

vim /etc/sysconfig/network-scripts/ifcfg-ens192

... 

BOOTPROTO=static
...
ONBOOT=yes
IPADDR=10.58.32.31
NETMASK=255.255.255.0

nmcli connection reload

基础配置

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 安装vim
yum install -y vim lrzsz

# 关闭selinux
sed -i 's/SELINUX=.*$/SELINUX=disabled/g'  /etc/selinux/config

# 关闭swap
swapoff -a 
sed -ri 's/.*swap.*/#&/' /etc/fstab

# 设置 open file
cat >> /etc/security/limits.conf << EOF

* hard noproc 65536
* soft noproc 65536
* hard nofile 65536
* soft nofile 65536
EOF

编辑hosts文件

cat >>  /etc/hosts <<EOF

10.58.32.30 k8s-master-lb
10.58.32.31 k8s-master01
10.58.32.32 k8s-master02
10.58.32.33 k8s-master03
10.58.32.34 k8s-node01
10.58.32.35 k8s-node02
EOF

ntp配置

yum install epel-release -y
yum -ivh https://mirrors.wlnmp.com/rockylinux/wlnmp-release-rocky-8.noarch.rpm
yum install wntp -y

ntpdate time2.aliyun.com

crontab -e
*/5 * * * * ntpdate time2.aliyun.com

cat >> /etc/rc.local << EOF

ntpdate time2.aliyun.com
EOF

chmod +x /etc/rc.local

系统优化

cat > /etc/sysctl.d/k8s_better.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

sysctl -p /etc/sysctl.d/k8s_better.conf

modprobe br_netfilter
lsmod |grep conntrack
modprobe ip_conntrack

安装IPVS

yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git

## 开启ipvs 转发
modprobe br_netfilter 

cat > /etc/sysconfig/modules/ipvs.modules << EOF 
#!/bin/bash 
modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_lc 
modprobe -- ip_vs_lblc 
modprobe -- ip_vs_dh
modprobe -- ip_vs_fo 
modprobe -- ip_vs_nq 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh 
modprobe -- ip_vs_sed 
modprobe -- ip_vs_ftp
modprobe -- nf_conntrack
modprobe -- ip_tables
modprobe -- ip_set
modprobe -- xt_set
modprobe -- ipt_set
modprobe -- ipt_rpfilter
modprobe -- ipt_REJECT
modprobe -- ipip
EOF 

chmod 755 /etc/sysconfig/modules/ipvs.modules 
bash /etc/sysconfig/modules/ipvs.modules 
lsmod | grep -e ip_vs -e nf_conntrack

特别注意，如果内核是4.19以下的，nf_conntrack 要改为nf_conntrack_ipv4

安装docker

yum install -y yum-utils device-mapper-persistent-data lvm2
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo 

yum list docker-ce --showduplicates
yum install docker-ce-19.03.13-3.el8

修改从cgroup方式

cat > /etc/docker/daemon.json <<EOF
{
    "registry-mirrors":[
        "https://registry.docker-cn.com",
        "https://hub-mirrors.c.163.com",
        "https://docker.mirrors.utsc.edu.cn"
    ],
    "exec-opts":["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 5,
    "log-opts":{
        "max-size": "300m",
        "max-file":"2"
    },
    "live-restore": true
}
EOF

然后

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

master01打通ssh到其他节点

master01节点执行

ssh-keygen

# 执行所有节点
for node in 10.58.32.{31..35};do
    ssh-copy-id root@${node}
done

下载k8s-server

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md #server-binaries

下载kubernetes-server-linux-amd64.tar.gz 并上传到所有节点

tar -xvf kubernetes-server-linux-amd64.tar.gz
mv kubernetes /

vim /etc/profile.d/k8s.sh

export K8S_HOME=/kubernetes
export K8S_SERVER=$K8S_HOME/server
export PATH=$PATH:$K8S_SERVER/bin

chmod +x /etc/profile.d/k8s.sh
ssource /etc/profile.d/k8s.sh

生成证书

master01节点下载cfssl

创建工作目录
mkdir /data/k8s-work
cd /data/k8s-work

下载地址： https://github.com/cloudflare/cfssl/releases?page=1
cfssl_1.6.3_linux_amd64
cfssljson_1.6.3_linux_amd64

移动到指定目录

chmod +x cfssl_1.6.3_linux_amd64 cfssljson_1.6.3_linux_amd64
cp cfssl_1.6.3_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.3_linux_amd64 /usr/local/bin/cfssljson

创建证书保存目录

# 所有节点
mkdir -p /etc/etcd/ssl
mkdir -p /etc/kubernetes/pki

下载开源工具包

git clone https://github.com/dotbalo/k8s-ha-install.git

cd k8s-ha-install
git branch -a
git checkout remotes/origin/manual-installation-v1.20.x

生成etcd证书

cd /root/k8s-ha-install/pki

[root@k8s-master01 pki]# ls
admin-csr.json ca-csr.json  front-proxy-ca-csr.json  kube-proxy-csr.json apiserver-csr.json  etcd-ca-csr.json  front-proxy-client-csr.json  manager-csr.json  ca-config.json etcd-csr.json  kubelet-csr.json  scheduler-csr.json

etcd的证书和k8s的证书是完全独立的，没有任何关系

CSR文件：证书签名请求文件

# ca证书
cfssl gencert -initca etcd-ca-csr.json|cfssljson -bare /etc/etcd/ssl/etcd-ca
# 客户端证书
cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,10.58.32.31,10.58.32.32,10.58.32.33 \
-profile=kubernetes \
etcd-csr.json|cfssljson -bare /etc/etcd/ssl/etcd

生成kubernetes证书

# ca证书
cfssl gencert -initca ca-csr.json|cfssljson -bare /etc/kubernetes/pki/ca

# apiserver需要证书
# 10.96.0是k8s service的网段，如果需要更改k8s service的网段，就需要更改10.96.0.1
# 10.58.32.30 vip网段
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-hostname=10.96.0.1,10.58.32.30,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.58.32.31,10.58.32.32,10.58.32.33 \
-profile=kubernetes \
apiserver-csr.json|cfssljson -bare /etc/kubernetes/pki/apiserver


# 生成controller-manage的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
manager-csr.json|cfssljson -bare /etc/kubernetes/pki/controller-manager

# 生成scheduler证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
scheduler-csr.json|cfssljson -bare /etc/kubernetes/pki/scheduler


# 生成admin的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
admin-csr.json|cfssljson -bare /etc/kubernetes/pki/admin

生成apiserver聚合证书

# ca证书
cfssl gencert -initca front-proxy-ca-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

# 生成聚合证书
# Requestheader-client-xxx    requestheader-allowed-xxx:aggerator
cfssl gencert \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
front-proxy-client-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-client

生成kubeconfig文件

生成controller的kubeconfig

# 设置集群项
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置用户项
kubectl config set-credentials system:kube-contoller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置环境项，上下文
kubectl config set-context system:kube-contoller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-contoller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 将上面的环境设置为默认环境
kubectl config use-context system:kube-contoller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

生成schedule的kubeconfig

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 设置用户项
kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 设置环境项，上下文
kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 将上面的环境设置为默认环境
kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

生成admin的kubeconfig

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 设置用户项
kubectl config set-credentials kubernetes-admin \
--client-certificate=/etc/kubernetes/pki/admin.pem \
--client-key=/etc/kubernetes/pki/admin-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 设置环境项，上下文
kubectl config set-context kubernetes-admin@kubernetes \
--cluster=kubernetes \
--user=kubernetes-admin \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

# 将上面的环境设置为默认环境
kubectl config use-context kubernetes-admin@kubernetes \
--kubeconfig=/etc/kubernetes/admin.kubeconfig

生成ServiceAccount key

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

拷贝证书到其他master节点

cd /etc/kubernetes/pki
scp -r ./* root@10.58.32.33:/etc/kubernetes/

配置etcd

一定要配置奇数个，否则会产生脑裂

一下操作master节点都要做

下载etcd

wget https://github.com/etcd-io/etcd/releases/download/v3.4.24/etcd-v3.4.24-linux-amd64.tar.gz

tar -xvf etcd-v3.4.24-linux-amd64.tar.gz
mv etcd-v3.4.24-linux-amd64 etcd-v3.4.24
mv etcd-v3.4.24 /kubernetes/

添加到PATH

vim /etc/profile.d/k8s.sh

export K8S_HOME=/kubernetes
export K8S_SERVER=$K8S_HOME/server
export PATH=$PATH:$K8S_SERVER/bin:$K8S_HOME/etcd-v3.4.24

source /etc/profile.d/k8s.sh

配置文件

配置文件每个节点多少有点不同，/etc/etcd/etcd.config.yaml

k8s-master01

name: 'k8s-master01'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.31:2380
listen-client-urls: https://10.58.32.31:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.31:2380
advertise-client-urls: https://10.58.32.31:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

k8s-master02

name: 'k8s-master02'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.32:2380
listen-client-urls: https://10.58.32.32:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.32:2380
advertise-client-urls: https://10.58.32.32:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

k8s-master03

name: 'k8s-master03'
data-dir: /kubernetes/etcd-v3.4.24/data
wal-dir: /kubernetes/etcd-v3.4.24/data/wal
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: https://10.58.32.33:2380
listen-client-urls: https://10.58.32.33:2379,http://127.0.0.1:2379
max-snapshots: 5
max-wals: 5
cors:
initial-advertise-peer-urls: https://10.58.32.33:2380
advertise-client-urls: https://10.58.32.33:2379
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.58.32.31:2380,k8s-master02=https://10.58.32.32:2380,k8s-master03=https://10.58.32.33:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  auto-tls: true
debug: false
logger: zap
log-outputs: [default]
force-new-cluster: false
auto-compaction-mode: periodic
auto-compaction-retention: "1"

启动system文件

cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/kubernetes/etcd-v3.4.24/etcd --config-file=/etc/etcd/etcd.config.yaml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

启动etcd集群

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

验证集群状态

ETCDCTL_API=3 /kubernetes/etcd-v3.4.24/etcdctl --write-out=table --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379 endpoint health

负载均衡器准备

安装haproxy与keepalived

yum -y install haproxy keepalived

HAProxy配置

cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  k8s-master01  10.58.32.31:6443 check
 server  k8s-master02  10.58.32.32:6443 check
 server  k8s-master03  10.58.32.33:6443 check
EOF

Keepalived

主从配置不一致，需要注意。

云上不支持keepalived

k8s-master01

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state MASTER
   interface ens192
   mcast_src_ip 10.58.32.31
   virtual_router_id 51
   priority 100
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.58.32.30
   }
   track_script {
      chk_apiserver
   }
}
EOF

k8s-master02，k8s-master02

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state BACKUP
   interface ens192
   mcast_src_ip 10.58.32.32,33 # 注意32,33是两个IP后缀
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.58.32.30
   }
   track_script {
      chk_apiserver
   }
}
EOF

健康检查脚本

master节点均要配置

cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi
EOF

chmod +x /etc/keepalived/check_apiserver.sh

启动服务并验证

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

ip address show

k8s组件安装

创建相关目录

mkdir -p /etc/kubernetes/manifests  /var/lib/kubelet /var/log/kubernetes

apiserver部署

该部署所有master节点都要执行

配置启动脚本

注意master节点的advertise-address不同，需要修改

vim /etc/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.service

[Service]
ExecStart=/kubernetes/server/bin/kube-apiserver \
  --v=2 \
  --logtostderr=true \
  --anonymous-auth=false \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --insecure-port=0 \
  --advertise-address=10.58.32.31 \
  --allow-privileged=true \
  --service-cluster-ip-range=10.96.0.0/16 \
  --service-node-port-range=30000-32767 \
  --etcd-servers=https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379 \
  --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --client-ca-file=/etc/kubernetes/pki/ca.pem \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --authorization-mode=Node,RBAC \
  --enable-bootstrap-token-auth \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --requestheader-allowed-names=aggregator \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
  --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem


Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

启动并验证

systemctl daemon-reload
systemctl enable --now kube-apiserver

systemctl status kube-apiserver

controller-manager部署

配置启动脚本

注意master节点的advertise-address不同，需要修改

vim /etc/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.service

[Service]
ExecStart=/kubernetes/server/bin/kube-controller-manager \
  --v=2 \
  --logtostderr=true \
  --bind-address=127.0.0.1 \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
  --leader-elect=true \
  --use-service-account-credentials=true \
  --node-monitor-grace-period=40s \
  --node-monitor-period=5s \
  --pod-eviction-timeout=2m0s \
  --controllers=*,bootstrapsigner,tokencleaner \
  --allocate-node-cidrs=true \
  --cluster-cidr=172.16.0.0/12 \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --node-cidr-mask-size=24 \
  --tls-cert-file=/etc/kubernetes/pki/controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/pki/controller-manager-key.pem


Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

schedule部署

cat >/etc/systemd/system/kube-scheduler.service<<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.service


[Service]
ExecStart=/kubernetes/server/bin/kube-scheduler \
	--v=2 \
	--address=127.0.0.1 \
	--leader-elect=true \
	--logtostderr=true \
	--kubeconfig=/etc/kubernetes/scheduler.kubeconfig


Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload 
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

TLS bootstrap自动颁发证书

注意： 如果不是高可用集群，vip要改成master01的IP，8443端口，改成默认的6443端口

要特别注意--token参数，是token_id.token_secret

自动给kubectl颁发的证书有效期是5年，到期会自动申请

cd /root/k8s-ha-install/bootstrap

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.58.32.30:8443 \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
--kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig

如果要修改bootstrap.secret.yaml的token-id和token-secret，要保证下面中的字符串保持一致，并且位数一样，还要保证上面命令中token信息和下面一致：

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c  # 注意这个
  token-secret: 2e4d610cf3e7426e  # 注意这个
  usage-bootstrap-authentication: "true"
... ...

然后配置kubectl的admin权限

mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

cd /root/k8s-ha-install/bootstrap
kubectl apply -f bootstrap.secret.yaml

secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

Node节点配置

拷贝证书

# node节点上
$ mkdir -p /etc/kubernetes/pki
$ mkdir -p /etc/etcd/ssl

# master01节点上
cd /etc/etcd/ssl
scp ./* root@10.58.32.34:/etc/etcd/ssl/
scp ./* root@10.58.32.35:/etc/etcd/ssl/
# master01节点上
for i in 10.58.32.{32..35};then do
  scp pki/ca.pem root@$i:/etc/kubernetes/pki/
  scp pki/ca-key.pem root@$i:/etc/kubernetes/pki
  scp pki/front-proxy-ca.pem root@$i:/etc/kubernetes/pki/
  scp kubelet-bootstrap.kubeconfig root@$i:/etc/kubernetes/
done

所有节点执行

mkdir -p /etc/kubernetes/manifests /var/lib/kubelet  /etc/systemd/system/kubelet.service.d /var/log/kubernetes

部署kubelet

vim /etc/kubernetes/kubelet-conf.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
  - 10.96.0.10
clusterDomain: cluster.local
containerLogMaxSize: 20Mi
containerLogMaxFiles: 5
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableDebuggingHandlers: true
enableControllerAttachDetach: true
enforceNodeAllocatable: 
- pods
eventRecordQPS: 5
eventBurst: 20
evictionHard:
  memory.available:  100Mi
  imagefs.available: 15%
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
fileCheckFrequency: 20s
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIQPS: 5
kubeAPIBurst: 10
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryPullQPS: 5
registryBurst: 10
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 10m
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionldleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

添加kubelet启动脚本

vim /etc/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/kubernetes/server/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=5


[Install]
WantedBy=multi-user.target

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig "
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin "
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yaml --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.4.1 "
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/kubernetes/server/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

然后

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

出现如下报错，并且有如下报错内容

Unable to register node "k8s-node02" with API server: Unauthorized

说明是认证的问题，重置TSL bootstrap有效

部署kube-proxy

只在master01节点执行就OK

注意：如果不是高可用集群，10.58.32.30:8443 的IP改成master01的IP，端口改为6443端口

kubectl -n kube-system create serviceaccount kube-proxy

kubectl create clusterrolebinding system:kube-proxy   \
--clusterrole system:node-proxier  \
--serviceaccount kube-system:kube-proxy

SECRET=$(kubectl -n kube-system get sa/kube-proxy \
    --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \
--output=jsonpath='{.data.token}' | base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes


kubectl config set-cluster kubernetes   \
--certificate-authority=/etc/kubernetes/pki/ca.pem  \
--embed-certs=true   \
--server=https://10.58.32.30:8443   \
--kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig

kubectl config set-credentials kubernetes  \
--token=${JWT_TOKEN}   \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig


kubectl config set-context kubernetes  \
--cluster=kubernetes  \
--user=kubernetes \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig


kubectl config use-context kubernetes  \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

然后切换目录

cd /root/k8s-ha-install/kube-proxy
ll 
-rw-r--r-- 1 root root  813 Mar 14 11:15 kube-proxy.conf
-rw-r--r-- 1 root root  288 Mar 14 11:15 kube-proxy.service
-rw-r--r-- 1 root root 3677 Mar 14 11:15 kube-proxy.yml

如果更改了集群Pod的网段，需要更改kube-proxy/kube-proxy.conf的clusterCIDR: 192.168.0.0/12参数为pod的网段（我们之前设置的）。

vim kube-proxy.conf

apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms

然后是kube-proxy.service，修改可执行文件地址为我们的地址

vim kube-proxy.service

[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/kubernetes/server/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.conf \
  --v=2

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

修改完毕，把他们发送到所有节点

for node in 10.58.32.{31..35};do
scp kube-proxy.conf root@$node:/etc/kubernetes/
scp kube-proxy.service root@$node:/etc/systemd/system/
scp /etc/kubernetes/kube-proxy.kubeconfig root@$node:/etc/kubernetes/
done

然后所有节点启动kube-proxy

systemctl daemon-reload && systemctl enable --now kube-proxy
systemctl status kube-proxy

部署calico

以下步骤只在master执行一次

cd /root/k8s-ha-install/calico

cp calico-etcd.yaml calico-etcd.yaml_backup

ETCD_CA=`cat /etc/etcd/ssl/etcd-ca.pem | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`

sed -i "s#.*etcd-ca:.*#  etcd-ca: ${ETCD_CA}#g" calico-etcd.yaml
sed -i "s#.*etcd-cert:.*#  etcd-cert: ${ETCD_CERT}#g" calico-etcd.yaml
sed -i "s#.*etcd-key:.*#  etcd-key: ${ETCD_KEY}#g" calico-etcd.yaml

sed -i 's#.*etcd_ca:.*#  etcd_ca: "/calico-secrets/etcd-ca"#g' calico-etcd.yaml
sed -i 's#.*etcd_cert:.*#  etcd_cert: "/calico-secrets/etcd-cert"#g' calico-etcd.yaml
sed -i 's#.*etcd_key:.*#  etcd_key: "/calico-secrets/etcd-key"#g' calico-etcd.yaml

sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-ca.pem#g" calico-etcd.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico-etcd.yaml
sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico-etcd.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico-etcd.yaml


$ vim calico-etcd.yaml 

注意修改如下部分：

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  etcd_endpoints: "https://10.58.32.31:2379,https://10.58.32.32:2379,https://10.58.32.33:2379"
  etcd_ca: ""
  etcd_cert: ""
  etcd_key: ""
   ... ...
  
  - name: CALICO_IPV4POOL_CIDR
    value: "172.16.0.0/12"  # pod的IP

出现如下报错，可解决

参考该博客解决问题： 简单部署3节点Kubernetes(k8s)集群之calico网络组件 - 简书 (jianshu.com)

部署CoreDNS

cd /root/k8s-ha-install/CoreDNS

把网关修改成我们自己的service的网关

vim coredns.yaml

spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10
  ports:
  - name: dns

然后安装CoreDNS

kubectl apply -f coredns.yaml

部署Metircs Server

cd  /root/k8s-ha-install/metrics-server-0.4.x
kubectl apply -f comp.yaml

然后查看状态

[root@k8s-master01 metrics-server-0.4.x]# kubectl get po -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
... ...
kube-system   metrics-server-595f65d8d5-txkkl            1/1     Running   0          103s

安装图形化工具

使用kuboard且使用docker话安装

Kuboard介绍 | Kuboard

demo命令(具体查看官方文档)：

docker run -d \
  --restart=unless-stopped \
  --name=kuboard \
  -p 80:80/tcp \
  -p 10081:10081/tcp \
  -e KUBOARD_ENDPOINT="http://10.58.32.32:80" \
  -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
  -v /kubernetes/kuboard-data:/data \
  eipwork/kuboard:v3

集群可用性测试

1. Pod 必须能解析 Servicee

2. Pod 必须能解析跨 namespace 的 Service

3. 每个节点都必须要能访问 Kubernetes 的kubermetes svc 443 和 kube-dns的 service 53

4. Pod 和Pod之前要能通

• 同namespace 能通信
• 跨 namespace 能通信
• 跨机器能通信

主机IP一定要用NAT模式，这样才能解析/etc/resolv.conf中的网关

问题可参考：

搭建Kubernetes集群时DNS无法解析问题的处理过程 - 掘金 (juejin.cn)

自定义 DNS 服务 | Kubernetes

调试 DNS 问题 | Kubernetes

kubectl exec busybox  -- nslookup kubernetes.default   # pod必须能解析service
kubectl exec busybox  -- nslookup kube-dns.kube-system # pod必须能解析跨namespace的service

telnet 10.96.0.1 443  # 对应说明3
telnet 10.96.0.10 53

# 对应第三点，进入一个pod，去ping其他机器或者pod
kubectl exec -it busybox -- sh   # 跨namespace/node
/ # ping 10.58.32.31

# 验证deployment部署
kubectl create deploy nginx-deployment --image=nginx --replicas=3
kubectl delete nginx-deployment

生产环境关键性配置

docker容器daemon.json

vim /etc/docker/daemon.json

{
    "registry-mirrors":[
        "https://registry.docker-cn.com",
        "https://hub-mirrors.c.163.com",
        "https://docker.mirrors.utsc.edu.cn"
    ],
    "exec-opts":["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10, //下载最大线程数
    "max-concurrent-uploads": 5, // 上传最大线程数
    "log-opts":{
        "max-size": "300m",
        "max-file":"2"
    },
    "live-restore": true    // 重启docker，保证运行中的容器不重启
}

controller-manager证书有效期

增长证书签发有效期

vim /etc/systemd/system/kube-controller-manager.service
# 添加
--experimental-cluster-signing-duration=87600h0m0s \

kubelet加密及资源配置

vim /etc/systemd/system/kube-kubelet.service
# 添加参数
# 增强kubelet加密强度
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
# 延长镜像下载时间（有些镜像下载很慢）
--image-pull-progress-deadline=30m

对应的配置文件：

allowdUnsafeSysctls 按需配置，会修改内核参数，

kubeReserved，systemReserved是给k8s和系统预留的资源

vim /etc/kubernetes/kubelet-config.yaml

# 添加如下配置
allowdUnsafeSysctls:
- "net.core*"
- "net.ipv4.*"
kubeReserved:
 cpu: "10m"
 memory: 1Gi
 ephemeral-storage: 10Gi
systemReserved:
 cpu: "1"
 memory: 1Gi
 ephemeral-storage: 10Gi

额外的说明

给master和worker节点设置roler

[root@k8s-master01 ~]# kubectl label nodes k8s-master02 node-role.kubernetes.io/master=''
node/k8s-master02 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-master03 node-role.kubernetes.io/master=''
node/k8s-master03 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-node01 node-role.kubernetes.io/woker=''
node/k8s-node01 labeled
[root@k8s-master01 ~]# kubectl label nodes k8s-node02 node-role.kubernetes.io/woker=''
node/k8s-node02 labeled


$ kubectl get node --show-labels

openssl 查看证书有效期

$ openssl x509 -in ca.pem -noout dates

TLS bootstrapping颁发续签/kubelet启动过程

官方文档：TLS 启动引导 | Kubernetes

1. Kubelet 启动

2. Kubelet 查看 kubelet.kubeconfig,文件，假设没有这个文件

3. Kubelet会查看本地的 bootstrap.kubeconfig

4. Kubelet读取 bootstrap.kubeconfig 文件，检索 apiserver 的 ur和一个 token

5. Kubelet链接 apiserver，使用这个 token 进行认证

   • Apiserver会识别 tokenid， apiserver 会查看该 tokenid 对无的bootstrap 的一个 secret
   • 找个这个secret 中的一个字段，apiserver ,把这个 token 识别成一个 username，名称是system:bootstrap:<token-id>，属于 system:bootstrappers 这个组，这个组具有申请 csr的权限，该组的权限绑定在一个叫system:nodebootstrapper的 custerroles
     • clusterrole k8s 集群级别的权限控制，它作用整个 k8s 集群
     • clusterrolebinding 集群权限的绑定，它可以帮某个clusterrole绑定到一个用户组或者 seviceaccount
   • CSR:相当于一个申请表，可以拿着这个申请表去申请我们的证书。

6. 经过上面的认证，kubelet 就有了一个创建和检索 CSR 的权限

7. Kubelet 为自己创建一个 CSR，名称为 kubernetesio/kube-apiserver-client-kubelet

8. CSR 被允许有两种方式:

   • K8s管理员使用 kubectl手动的颁发证书
   • 如果配置了相关权限，kube-controller-manager 会自动同意。
     • Controller-manager 有一个CSRApprovingContraller。他会校验 kubelet发来的 csr的 username和 group 是否有创建 csr的权限，而且还要验证签发着是否是kubernetes.io/kube-apiserver-client-kubelet
     • Controller-manager 同意 CSR 请求

9. CSR 被同意后，controller-manager 创建 kubelet 的证书文件

10. Controller-manager 将证书更新至 CSr的 status 字段

11. Kubelet 从apiserver 获取证书

12. Kubelet 从获取到的 key 和证书文件创建 kubelet.kubeconfig

13. Kubelet 启动完成并正常工作

14. 可选:如果配置了自动续期，kubelet 会在证书文件过期的时候利用之前的 kubeconfig 文件去申请一个新的证书，相当于续约。

15. 新的证书被同意或签发，取决于我们的配置

    • Kubelet 创建的 CSR 是属于一个O（orgian）: system:nodes
    • CN: system:nodes:主机名

添加默认路由

chmod +x /etc/rc.local

vim /etc/rc.local
route add default gw 10.58.32.2 ens192

resolv.conf不断被修改

解决方法
/etc/sysconfig/network-scripts/ifcfg-eth0或者自己网卡配置文件增加一行：PEERDNS=no，然后重新启动网络即可。修改后，重启主机等操作便不会使/etc/resolv.conf被dhclient修改

# Generated by NetworkManager
search localdomain
nameserver 10.58.32.2

关于service注册服务

Linux配置service服务 - 裸睡的猪 - 博客园 (cnblogs.com)

Linux下服务的管理_vendor preset_liuchonghua的博客-CSDN博客

Systemd 入门教程：命令篇 - 阮一峰的网络日志 (ruanyifeng.com)

多网卡集群修改calico通信网卡

vim calico.yaml

env:
  ... ...
  - name: IP_AUTODETECTION_METHOD
    value: "interface=ens33"
  ... ...