keepalived实现nginx高可用（keepalived 2.2.7 + nginx 1.20.2）

Linux系统-部署-运维系列导航

 

一、架构介绍

负载均衡器的高可用方案架构如下，可以适用多种业务场景。
  • 负载均衡方案可以是lvs、nginx、haproxy等
  • 业务服务器可以是web服务器如nginx、tomcat、apache、iis等，也可以是java、.ne、mysqlt等tcp应用服务

 

二、架构搭建

本文将要实现的架构设计如下
机器名称  
IP
服务器角色
备注
localhost
192.168.11.10
keepalived master
nginx负载均衡
keepalived 2.2.7 + nginx 1.20.2
localhost
192.168.11.11
keepalived master
nginx负载均衡
keepalived 2.2.7 + nginx 1.20.2
localhost
192.168.11.13
nginx web服务器
nginx 1.20.2
localhost
192.168.11.14
nginx web服务器
nginx 1.20.2
keepalived实现nginx高可用，包括以下步骤
  1. nginx web服务搭建
  2. nginx负载均衡搭建
  3. keepalived安装与配置
  4. keepalived运行测试

 

1.nginx web服务搭建

使用nginx默认 html 页面演示，在11.13、11.14服务器上分别执行以下步骤
1.1nginx安装
 
1.2修改nginx默认页面，显示服务器IP
[root@localhost local]# cd /usr/local/nginx/
[root@localhost nginx]# vim html/index.html

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<h1>192.168.11.13</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

 

1.3配置nginx，监听8000端口
worker_processes  2;

events {
    use epoll;
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    
    server {
        listen       8000;
        server_name  localhost;

        location / {
            root   html;
            index  index.html index.htm;
        }
        
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

 

1.4nginx测试

 

2.nginx负载均衡搭建

在11.10、11.10服务器上分别执行以下步骤搭建nginx负载均衡，负载服务器为11.13、11.14
 
2.1nginx安装

 

2.2负载均衡配置，其中负载均衡器监听 80 端口
worker_processes  2;

events {
    use epoll;
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;

    upstream local_nginx {
        server 192.168.11.13:8000 weight=1;
        server 192.168.11.14:8000 weight=2;
    }

    server {
        listen       80;
        server_name  192.168.11.13;
        
        location / {
            root   html;
            index  index.html index.htm;
            proxy_pass http://local_nginx;
        }
        
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

 

2.3负载均衡测试
由于11.14配置权重weight=2，所以每2次请求代理到11.14，1次请求代理到11.13

 

3.keepalived安装与配置

3.1keepalived安装

 

3.2keepalived配置，默认11.10为master，配置priority=200，11.11则priority=100
特别关注：配置文件中指定的所有路径，请在启动前确保已存在
! Configuration File for keepalived

global_defs {
   notification_email {
     
   }   
   #keepalived机器标识，无特殊作用，一般为机器名
   router_id ha_nginx
}

#检查脚本，可以用来关联业务，脚本执行结果决定是否准备切换
#权重策略是：根据脚本执行结果计算权重，然后触发keepalived重新选举
#当weight > 0时：脚本执行成功了 Priority + Weight，执行失败 Priority
#当weight < 0时：脚本执行成功了 Priority 执行失败 Priority + Weight
vrrp_script ha_nginx{
    #脚本路径，脚本执行是否成功，根据脚本的退出码确认，默认为0，即exit 0
    script "/usr/local/ha_nginx/ha_nginx.sh"
    #脚本检测周期，单位秒
    interval 2
    #权重，本方案通过重启keepalived服务触发切换，不配置权重策略
    #weight 10
}

#VRRP协议配置
vrrp_instance VI_1 {
    #集群初始状态统一配置为 BACKUP，当至少2台keepalived启动后，将根据priority重新竞选角色
    state BACKUP
    interface enp0s3
    #虚拟路由id，同一个集群中的keepalived设置一致
    virtual_router_id 100
    #优先级决定最终的master角色
    priority 200
    #不抢占，即异常恢复后，不立即抢占master角色
    nopreempt
    #主备之间通信检查的时间间隔，单位秒
    advert_int 1
    authentication {
    #keepalived之间认证类型为密码
        auth_type PASS
        auth_pass 1234
    }
    #虚拟IP池
    virtual_ipaddress {
        #VIP地址，一行一个，格式为 <IP地址>/<掩码> brd <广播地址> dev <网卡设备> scope <范围如global> label <网卡别名>
        192.168.11.100/24
    }

    #检查脚本，与vrrp_script对应
    track_script{
        ha_nginx
    }
}

 

ha_nginx.sh脚本
配置脚本可执行权限
[root@localhost ~]# chmod +x /usr/local/ha_nginx/ha_nginx.sh

 

脚本内容
#!/bin/bash
#监控日志
source /etc/profile
monitorLogPath=/usr/local/ha_nginx/monitor.log
touch $monitorLogPath

#格式化日期时间
function getDatetime(){
    local cur=`date "+%Y-%m-%d %H:%M:%S"`
     echo $cur  
}

#检测nginx进程
function checkNginxProcess(){
    #检测nginx服务，可以使用进程，也可以使用端口
    #此处检测nginx进程
    local num=`ps -C nginx --no-header | wc -l`
   #local num=`netstat -lntup | grep -w 8099 | wc -l`
    if [ $num -eq 0 ];then
        echo 0
    else
        echo 1
    fi
}

#定义变量，nginx是否运行
run=`checkNginxProcess`
if [ $run -eq 0 ];then
    #nginx异常，先启动nginx
    echo `getDatetime` "nginx error，start nginx" >> $monitorLogPath 
    nginx -c /usr/local/nginx/conf/nginx.conf
    run=`checkNginxProcess`
    if [ $run -eq 0 ];then
        #启动nginx失败，停止keepalived服务，停止热备，触发keepalived切换
        echo `getDatetime` "start nginx failed, kill keepalived" >> $monitorLogPath
        service keepalived restart
    else            
        #keepalived切换，因为nginx运行异常，则说明服务器稳定性差
        echo `getDatetime` "restart keepalived" >> $monitorLogPath
        service keepalived restart
    fi
#else
#    echo `getDatetime` "nginx is alive" >> $monitorLogPath
fi

 

三、架构测试

1.确认keepalived master
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:80:14:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.10/24 brd 192.168.11.255 scope global noprefixroute enp0s3
       valid_lft forever preferred_lft forever
    inet 192.168.11.100/24 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::a33a:d49b:da44:119a/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::c08b:489f:1587:3bb6/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

 

2.测试master
当前VIP绑定在11.10，访问测试

 

3.模拟11.10负载均衡器故障
[root@localhost ~]# nginx -s stop
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:80:14:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.10/24 brd 192.168.11.255 scope global noprefixroute enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::a33a:d49b:da44:119a/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::c08b:489f:1587:3bb6/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

 

查看监控日志 /usr/local/ha_nginx/monitor.log
2022-01-29 14:48:56 nginx error，start nginx
2022-01-29 14:48:56 restart keepalived

 

4.确认VIP已经漂移
在11.11服务器查看
[root@localhost nginx]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:1e:c1:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.11/24 brd 192.168.11.255 scope global noprefixroute enp0s3
       valid_lft forever preferred_lft forever
    inet 192.168.11.100/24 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::a33a:d49b:da44:119a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

 

业务访问正常

 

四、后记

关于keepalived VIP漂移
nginx服务异常，则该服务器可能存在性能问题，应该主动触发keepalived VIP漂移（通过重启keepalived服务），并自动重启nginx。如果keepalived重启后VIP重新漂移至该节点，尽管VIP漂移间隔很小，但客户端依然可能受到影响。
为了避免出现此问题，本文中将所有 keepalived 实例设置为 backup 角色，同时添加了 nopreempt 配置项，即设置为 非抢占 模式，如此，nginx与keepalived服务重启后，不会主动竞争 master，客户端业务保持正常。
 
特别关注：系统默认启用了SELinux内核模块（安全子系统），所以在服务绑定/监听某些端口时，提示无访问权限，此时需要禁用SELinux，修改 /etc/selinux/config 文件，设置SELINUX=disabled
Can't start server: Bind on TCP/IP port: Permission denied

 

特别关注：selinux设置完成需要重启生效，如果当前不方便重启，可以执行 setenforce 0 临时关闭selinux，下次重启是配置再生效
 
特别关注：系统默认启用了防火墙，请在启动服务前关闭防火墙，或在防火墙中添加服务端口
 
posted on 2023-09-05 09:39  xiaoyaozhe  阅读(111)  评论(0)  编辑  收藏  举报
Copyright © 2024 xiaoyaozhe
Powered by .NET 9.0 on Kubernetes Powered by: 博客园