2-3.4 配置nginx四层反向代理

3.4.1 集群规划#

主机名	角色	IP地址
pg60-11.k8s.host.com	4层负载均衡	10.20.60.11
pg60-12.k8s.host.com	4层负载均衡	10.20.60.12
注意：这里 10.20.60.11 和 10.20.60.12 使用nginx做4层负载均衡器，在keepalived上配置vip：10.20.60.10，代理两个 kube-apiserver，实现高可用。

3.4.2 编译安装Nginx#

详见《源码安装openresty-1.11.2.5》

3.4.3 配置nginx vhost#

Copy
shell> cat /opt/openresty/nginx/conf/nginx.conf 
stream {
    upstream kube-apiserver {
        server 10.20.60.21:6443 max_fails=3 fail_timeout=30s;
        server 10.20.60.22:6443 max_fails=3 fail_timeout=30s;
        server 10.20.60.23:6443 max_fails=3 fail_timeout=30s;
    }
    server {
        listen 6443;
        proxy_connect_timeout 2s;
        proxy_timeout 900s;
        proxy_pass kube-apiserver;
    }
}

3.4.4 安装与配置keepalived#

• 安装keepalived

Copy
shell> yum -y install keepalived

• 创建keepalived检查端口脚本check_port.sh

Copy
shell> cat /etc/keepalived/check_port.sh 
#!/bin/bash
#keepalived 监控端口脚本
#使用方法：
#在keepalived的配置文件中
#vrrp_script check_port {#创建一个vrrp_script脚本,检查配置
# script "/etc/keepalived/check_port.sh 6379" #配置监听的端口
# interval 2 #检查脚本的频率,单位（秒）
#}
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
else
        echo "Check Port Cant Be Empty!"
fi

• 配置keepalived（主）

Copy
shell> cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id 10.20.60.11
}

vrrp_script chk_nginx {
   script "/etc/keepalived/check_port.sh 6443"
   interval 2
   weight -20
}

vrrp_instance VI_1 {
   state MASTER
   interface eth0
   virtual_router_id 51
   priority 150
   advert_int 1
   mcast_src_ip 10.20.60.11
   nopreempt
   authentication {
      auth_type PASS
      auth_pass 11111111
   }
   track_script {
      chk_nginx
   }
   virtual_ipaddress {
      10.20.60.10
   }
}

• 配置keepalived（备）

Copy
shell> cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id 10.20.60.12
}

vrrp_script chk_nginx {
   script "/etc/keepalived/check_port.sh 6443"
   interval 2
   weight -20
}

vrrp_instance VI_1 {
   state BACKUP
   interface eth0
   virtual_router_id 51
   priority 100
   advert_int 1
   mcast_src_ip 10.20.60.12
   authentication {
      auth_type PASS
      auth_pass 11111111
   }
   track_script {
      chk_nginx
   }
   virtual_ipaddress {
      10.20.60.10
   }
}

3.4.5 启动并检查服务#

Copy
shell> systemctl start keepalived
shell> systemctl enable keepalieve
shell> /opt/openresty/nginx/sbin/nginx -s reload