本地节点：
主机名 IP 规格 说明
k8s-mainnode-01 192.168.1.15 ip11号机上的 虚拟机2c2gb k8sMaster1、Haproxy备1、keepalivedVip192.168.1.8备
k8s-mainnode-02 192.168.1.14 ip10号机上的 虚拟机2c2gb k8sMaster2、Haproxy备2、keepalivedVip192.168.1.8备
k8s-mainnode-03 192.168.1.13 物理机8c16gb k8sMaster3、k8sWork0、Haproxy主、keepalivedVip192.168.1.8主
sqlnode-01 192.168.1.12 物理机4c16gb k8sWorker1
sqlnode-02 192.168.1.11 物理机8c32gb k8sWorker2
sqlnode-03 192.168.1.10 物理机8c32gb k8sWorker3
sqlnode-04 192.168.1.16 主笔记本ip22号机上的 虚拟机6c16gb k8sWorker4
sqlnode-05 192.168.1.17 主笔记本ip22号机上的 虚拟机6c16gb k8sWorker5
sqlnode-06 192.168.1.18 主笔记本ip22号机上的 虚拟机6c16gb k8sWorker6
整3个主节点是因为自己的服务器集群环境没有机房那么稳定；
修改所有节点
cat >>/etc/hosts <<EOF
192.168.1.15 k8s-mainnode-01
192.168.1.14 k8s-mainnode-02
192.168.1.13 k8s-mainnode-03
192.168.1.12 sqlnode-01
192.168.1.11 sqlnode-02
192.168.1.10 sqlnode-03
192.168.1.16 sqlnode-04
192.168.1.17 sqlnode-05
192.168.1.18 sqlnode-06
EOF
按照hosts文件的ip和主机名对应关系，使用下面的命令重新设定主机名
hostnamectl set-hostname sqlnode-02
集群规划
计划部署“1负载均衡集群（1主2备基于keepalived的vip） + 3 Master节点 + 7 Worker节点”的高可用Kubernetes集群：
部署负载均衡服务器
先配一个nginx，在k8s的6443端口没启用一前先替代一下，以便不止前期的负载均衡和高可用；
在3台k8s main node安装nginx：
mkdir -p /opt/nginx-src
cd /opt/nginx-src
yum install -y gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel axel
axel -n 5 https://nginx.org/download/nginx-1.20.1.tar.gz
tar -zxvf nginx-1.20.1.tar.gz
cd nginx-1.20.1
./configure
make && make install
/usr/local/nginx/sbin/nginx #因为仅仅为了测试，所以只需要临时启动一下其实就可以了；
修改一下每台机器上的/usr/local/nginx/html/index.html，以便区别各个节点的不同；
在3台k8s main node安装keepalived：
mkdir -p /opt/keepalived
cd /opt/keepalived
rm -rf *
wget https://www.keepalived.org/software/keepalived-2.2.4.tar.gz
tar -zxvf keepalived-2.2.4.tar.gz
cd keepalived-2.2.4
./configure && make && make install
在keepalived的主节点ip13号机：
cat > /etc/rc.d/init.d/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id 13
}
vrrp_instance VI_1 {
state MASTER
interface enp0s20u1u1
virtual_router_id 3
mcast_src_ip 192.168.1.13
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.1.8/24
}
}
EOF
在keepalived的主节点ip13号机：
cat > /etc/rc.d/init.d/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id 13
}
vrrp_instance VI_1 {
state MASTER
interface enp0s20u1u1
virtual_router_id 3
mcast_src_ip 192.168.1.13
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.1.8/24
}
}
EOF
在从节点ip14号机：
cat > /etc/rc.d/init.d/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id 14
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 3
mcast_src_ip 192.168.1.14
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.1.8/24
}
}
EOF
在从节点ip15号机：
cat > /etc/rc.d/init.d/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id 15
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 3
mcast_src_ip 192.168.1.15
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.1.8/24
}
}
EOF
cp -a /usr/local/etc/keepalived /etc/init.d/ cp -a /usr/local/etc/sysconfig/keepalived /etc/sysconfig/ cp -a /usr/local/sbin/keepalived /usr/sbin/
mkdir -p /etc/keepalived
ln -s /etc/rc.d/init.d/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
systemctl daemon-reload
systemctl start keepalived
systemctl enable keepalived
在3台k8s main node安装HAProxy：
yum install haproxy -y
#注意可能要关闭senlinux，
1、临时关闭：
setenforce 0
2、修改配置文件需要重启机器：
修改/etc/selinux/config 文件
将SELINUX=enforcing改为SELINUX=disabled
可能要注释掉option forwardfor
在/etc/haproxy/haproxy.cfg尾部增加
vi /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
bind 192.168.1.8:6443
mode tcp
option tcplog
default_backend apiserver
#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
option httpchk GET /healthz #用nginx测试harpoxy的时候要注释掉这条
http-check expect status 200 #用nginx测试harpoxy的时候要注释掉这条
mode tcp
option ssl-hello-chk #用nginx测试harpoxy的时候要注释掉这条
balance roundrobin
server k8s-mainnode-01 192.168.1.15:6443 check
server k8s-mainnode-02 192.168.1.14:6443 check
server k8s-mainnode-03 192.168.1.13:6443 check backup #加备是因为这个节点我打算也做work节点，因为主要做work节点所以不打算他多参与这里的事情
在keepalived的备机上如果监听192.168.1.8，默认会报错，
解决方法：
在所有haproxy的节点上，在/etc/sysctl.conf添加
cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_nonlocal_bind = 1 #忽略监听ip的检查
EOF
sysctl -p
以服务方式运行HAProxy：
systemctl daemon-reload
systemctl enable haproxy
systemctl start haproxy
待续。。。
