nginx+keepalived+k8s

一.nginx的安装

1.nginx安装包下载

在官网 https://nginx.org/en/download.html下载linux的tar包选择合适的版本如https://nginx.org/download/nginx-1.24.0.tar.gz

2.安装依赖

yum install gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel

3.安装nginx

tar -xvf nginx-1.24.0.tar.gz
cd nginx-1.24.0 2
./configure --prefix=/data/nginx  --with-http_stub_status_module --with-http_ssl_module  --with-stream
make && make install

4.修改index.html

方便后面keepalived的测试

nginx1
vim /data/nginx/html/index.html

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>My Website</title>
</head>
<body>
    <h1>Welcome to nginx1!</h1>
    <p>Current time: <span id="current-time"></span></p>

    <script>
        // 获取当前时间并更新页面
        function updateTime() {
            var currentTime = new Date();
            var currentDateString = currentTime.toLocaleString();
            document.getElementById("current-time").innerHTML = currentDateString;
        }

        // 每秒钟更新一次时间
        setInterval(updateTime, 1000);
    </script>
</body>
</html>


nginx2
vim /data/nginx/html/index.html

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>My Website</title>
</head>
<body>
    <h1>Welcome to nginx2!</h1>
    <p>Current time: <span id="current-time"></span></p>

    <script>
        // 获取当前时间并更新页面
        function updateTime() {
            var currentTime = new Date();
            var currentDateString = currentTime.toLocaleString();
            document.getElementById("current-time").innerHTML = currentDateString;
        }

        // 每秒钟更新一次时间
        setInterval(updateTime, 1000);
    </script>
</body>
</html>

显示效果如下 server名+时间的显示

二.keepalive的安装与配置

1.使用yum安装

yum install keepalived -y

2.修改配置文件

1）需要在global_defs 中添加

script_user root

enable_script_security
否则会报警告：WARNING -default user ‘keepalived_script’ for script execution does not exist -please create. Mar 26 11:37:09 localhost.localdomain Keepalived_vrrp[4587]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.

2）需要将vrrp_strict注释掉，否则会ping不通vip

3）需要将原配置文件中所有的virtual_server都删除，否则vip指不到nginx，因为virtual_server中的lb_kind NAT模式不支持域内访问。

keepalived1
vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   script_user root
   enable_script_security
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_script nginx_check {
        script "/etc/keepalived/nginx_health.sh"
        interval 2
        weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.38.0.144
    }
   track_script {
        nginx_check
    }
}

keepalived
vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   script_user root
   enable_script_security
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_script nginx_check {
        script "/etc/keepalived/nginx_health.sh"
        interval 2
        weight -20
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.38.0.144
    }
     track_script {
        nginx_check
    }
}

3.测试

1）将nginx1 停掉,由于/etc/keepalived/nginx_health.sh文件，nginx会马上自动重启

systemctl stop nginx
systemctl status nginx

2）将keepalived1停掉，vip会飘到nginx2

keepalived测试完成。

三.nginx配置k8s服务器的转发

1.配置文件

其中kube-api端口644使用stream tcp进行转发。80和443也需要用stream tcp进行转发，否则在外部nginx或者vip上解析域名不可以解析到k8s的对应应用中。（这种说法是错的：80 和443才有http模块进行转发，使用http模块进行转发可以设置各种http参数，否则只在ingress里设置会不生效，因为用户访问的是vip的nginx，所以请求是发给这个nginx而不是发给ingress的，但是ingress里的各种http参数生不生效这个还未经测试）

两台nginx服务器都使用如下配置文件。

vim /data/nginx/conf/nginx.conf

#user nobody;
worker_processes auto;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
worker_connections 1024;
}

stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

access_log logs/k8s-access.log main;

upstream k8s-http {
server 10.12.3.136:80;
server 10.12.3.141:80;
server 10.12.3.142:80;
server 10.12.3.143:80;
server 10.12.3.144:80;
}
upstream k8s-https {
server 10.12.3.136:443;
server 10.12.3.141:443;
server 10.12.3.142:443;
server 10.12.3.143:443;
server 10.12.3.144:443;
}
upstream k8s-apiserver {
server 10.12.3.142:6443;
server 10.12.3.143:6443;
server 10.12.3.144:6443;
}
server {
listen 80;
proxy_pass k8s-http;
}
server {
listen 443;
proxy_pass k8s-https;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}