二进制部署k8s集群v1.23.9版本-5-安装apiserver集群

5.1、安装docker环境

192.168.1.200/201/202/203四台服务器操作

5.1.1、安装docker

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

5.1.2、配置docker

mkdir -p /etc/docker/
vim /etc/docker/daemon.json
[root@hfqg1-201 web]# cat /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com","quay.io","harbor.qgutech.com"],
"registry-mirrors": ["https://6su5l99j.mirror.aliyuncs.com"],
"bip": "172.1.201.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}

说明：

• exec-opts：CPU/MEM的资源管理方式
• registry-mirrors：镜像源
• insecure-registries：信任的HTTP镜像仓库
• bip根据不同的主机修改：
• 201：172.1.201.1/24
• 202：172.1.202.1/24
• 203：172.1.203.1/24
• 200：172.1.200.1/24

创建目录
mkdir -pv /data/docker

5.1.3、启动docker

systemctl enable docker --now
systemctl status docker -l

docker info

docker version

5.2、部署kube-apiserver集群

5.2.1、集群规划

主机名	角色	IP
hfqg1-201	kube-apiserver	192.168.1.201
hfqg1-202	kube-apiserver	192.168.1.202
hfqg1-203	kube-apiserver	192.168.1.203
hfqg1-198	nginx+keepalived	192.168.1.198
hfqg1-199	nginx+keepalived	192.168.1.199
无	VIP	192.168.1.196

注意：这里192.168.1.198和192.168.1.199使用nginx做4层负载均衡，用keepalived跑一个vip：192.168.1.96，代理三个kube-apiserver，实现高可用

5.2.2、下载软件、解压、做软链接

在192.168.1.201服务器上操作

cd /opt/src
wget https://dl.k8s.io/v1.23.9/kubernetes-server-linux-amd64.tar.gz
tar xf kubernetes-server-linux-amd64.tar.gz -C /opt/
mv kubernetes kubernetes-v1.23.9
ln -s kubernetes-v1.23.9 kubernetes
cd kubernetes
rm -f kubernetes-src.tar.gz
cd /opt/src/kubernetes/server/bin
rm -f *.tar
rm -f *_tag

5.2.3、签发kube-apiserver证书

192.168.1.200服务器上操作

cd /opt/certs/
[root@hfqg1-200 certs]# cat kube-apiserver-csr.json
{
"CN": "kube-apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"192.168.1.196",
"192.168.1.197",
"192.168.1.198",
"192.168.1.199",
"192.168.1.200",
"192.168.1.201",
"192.168.1.202",
"192.168.1.203"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "system:masters",
"OU": "system"
}
]
}

说明：

• CN：K8S会提取CN字段的值作为用户名，实际是指K8S的"RoleBinding/ClusterRoleBinding"资源中，“subjects.kind”的值为“User"，
• hosts：包括所有Master节点的IP地址，LB节点、LB集群节点、ClusterIP的首个IP，K8S的“ClusterIP”的范围在“--service-cluster-ip-range”中指定，取值为192.168.0.0/16，此处配置为192.168.0.1
• names
• C：CN
• ST：
• L：
• O：“system:masters”，定义“O”值的原因：apiserver向kubelet发起请求时，将复用此证书，参看官方文档。K8S默认会提取“O”字段的值作为组，这实际是指K8S的“RoleBinding/ClusterRoleBinding”资源中"subjects.kind"的值为“Group”，

生成证书

cfssl gencert
-ca=ca.pem
-ca-key=ca-key.pem
-config=ca-config.json
-profile=kubernetes
kube-apiserver-csr.json | cfssl-json -bare kube-apiserver

ll kube-apiserver*

5.2.4、拷贝证书到运算节点

192.168.1.201服务器操作
把6张证书和密钥从200主机上拷贝到certs目录下
cd /opt/kubernetes/server/bin/certs

5.2.5、生成token.csv文件

192.168.1.201服务器操作
该文件的作用是，在工作节点（kubelet）加入K8S集群的过程中，向kube-apiserver申请签发证书。
cd /opt/kubernetes/server/bin/certs
cat > kube-apiserver.token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
[root@hfqg1-201 certs]# cat kube-apiserver.token.csv
8479def061dc3709917faec5abc618a9,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

5.2.6、创建启动脚本

192.168.1.201服务器操作

cd /opt/kubernetes/server/bin
[root@hfqg1-201 bin]# cat kube-apiserver.sh
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/kubernetes/server/bin/kube-apiserver
--runtime-config=api/all=true
--anonymous-auth=false
--v=2
--allow-privileged=true
--bind-address=0.0.0.0
--secure-port=6443
--advertise-address=192.168.1.201
--service-cluster-ip-range=192.168.0.0/16
--service-node-port-range=3000-29999
--audit-log-path=/data/logs/kubernetes/kube-apiserver/audit-log
--authorization-mode=Node,RBAC
--client-ca-file=/opt/kubernetes/server/bin/certs/ca.pem
--etcd-cafile=/opt/kubernetes/server/bin/certs/ca.pem
--etcd-certfile=/opt/kubernetes/server/bin/certs/etcd.pem
--etcd-keyfile=/opt/kubernetes/server/bin/certs/etcd-key.pem
--etcd-servers=https://192.168.1.201:2379,https://192.168.1.202:2379,https://192.168.1.203:2379
--service-account-key-file=/opt/kubernetes/server/bin/certs/ca.pem
--service-account-signing-key-file=/opt/kubernetes/server/bin/certs/ca-key.pem
--service-account-issuer=https://kubernetes.default.svc.cluster.local
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
--kubelet-client-certificate=/opt/kubernetes/server/bin/certs/kube-apiserver.pem
--kubelet-client-key=/opt/kubernetes/server/bin/certs/kube-apiserver-key.pem
--enable-bootstrap-token-auth=true
--log-dir=/data/logs/kubernetes/kube-apiserver
--tls-cert-file=/opt/kubernetes/server/bin/certs/kube-apiserver.pem
--tls-private-key-file=/opt/kubernetes/server/bin/certs/kube-apiserver-key.pem
--enable-aggregator-routing=true
--token-auth-file=/opt/kubernetes/server/bin/certs/kube-apiserver.token.csv

调整权限和目录
chmod +x kube-apiserver.sh
mkdir -pv /data/logs/kubernetes/kube-apiserver

5.2.7、创建supervisor配置

192.168.1.201服务器操作

[root@hfqg1-201 kube-apiserver]# cat /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver-1-201]
command=/opt/kubernetes/server/bin/kube-apiserver.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

5.2.8、启动服务并检查

supervisorctl update
supervisorctl status

netstat -lntp|grep kube-api

5.2.9、安装部署其它两个节点

192.168.1.201服务器操作
cd /opt/
scp -r kubernetes-v1.23.9/ hfqg1-202:/opt/
scp -r kubernetes-v1.23.9/ hfqg1-203:/opt/
192.168.1.202/203操作
创建目录
mkdir -pv /data/logs/kubernetes/kube-apiserver
做软链接
cd /opt
ln -s kubernetes-v1.23.9 kubernetes
修改kube-apiserver-startup.sh中IP地址
202如下图所示

203如下图所示

修改 /etc/supervisord.d/kube-apiserver.ini 中的名称
202如下图所示

203如下图所示

启动kube-apiserver服务
supervisorctl update
supervisorctl status
检查集群监控状态
curl --insecure https://192.168.1.201:6443/
curl --insecure https://192.168.1.202:6443/
curl --insecure https://192.168.1.203:6443/

至此，kube-apiserver安装完成。

5.3、配置四层反向代理

5.3.1、apiserver监听端口

netstat -lntp|grep api

5.3.2、说明

用keepalived跑一个192.168.1.196的vip
用192.168.1.196上的8443，反向代理192.168.1.201、192.168.1.202和192.168.1.203上的6443端口。

下面的操作在192.168.1.198和192.168.1.199主机上进行。

5.3.3、安装nginx

yum install nginx nginx-mod-stream -y
配置nginx

vim /etc/nginx/nginx.conf
stream {
log_format proxy '$time_local|$remote_addr|$upstream_addr|$protocol|$status|'
'$session_time|$upstream_connect_time|$bytes_sent|$bytes_received|'
'$upstream_bytes_sent|$upstream_bytes_received' ;

upstream kube-apiserver {
server 192.168.1.201:6443 max_fails=3 fail_timeout=30s;
server 192.168.1.202:6443 max_fails=3 fail_timeout=30s;
server 192.168.1.203:6443 max_fails=3 fail_timeout=30s;
}
server {
listen 8443;
proxy_connect_timeout 2s;
proxy_timeout 900s;
proxy_pass kube-apiserver;
access_log /var/log/nginx/proxy.log proxy;
}
}

检查语法
nginx -t

启动nginx
systemctl enable nginx --now
netstat -luntp | grep nginx

5.3.4、安装keepalived

yum install keepalived -y
198服务器

vim /etc/keepalived/check_apiserver.sh
[root@hfqg1-198 ~]# cat /etc/keepalived/check_apiserver.sh
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep nginx)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi

199服务器

[root@hfqg1-199 ~]# cat /etc/keepalived/check_apiserver.sh
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep nginx)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi

增加可执行权限
chmod +x /etc/keepalived/check_apiserver.sh

keepalived主配置

198服务器
[root@hfqg1-198 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface ens192
virtual_router_id 251
priority 100
advert_int 2
mcast_src_ip 192.168.1.198
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_apiserver
}
virtual_ipaddress {
192.168.1.196
}
}

199服务器
[root@hfqg1-199 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens192
virtual_router_id 251
priority 90
advert_int 2
mcast_src_ip 192.168.1.199
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_apiserver
}
virtual_ipaddress {
192.168.1.196
}
}

启动keepalived服务
systemctl enable keepalived --now
ip addr

高可用测试：

• 在198机器上停止nginx，vip不存在了，199机器查看，发现vip跑到199机器上了。
• 重启nginx，vip还在198主机上。
• 这是因为在198上配置了 nopreempt ，设置非抢占式，当主服务down，vip漂移到备机，当主机服务up，vip依然在备机上。
• 如果要想vip回到198上，重新启动keepalived。。。