nginx + keepalived 配置高可用nginx负载均衡集群

记录下搭建高可用的nginx负载均衡集群，nginx和keepalived大的配置都属于最简配置。
nginx官网：https://nginx.org/en/docs/
keepalived的配置详解，官网地址： https://keepalived.org/manpage.html

环境准备：至少3个节点虚拟机。（可以将web1和web2部署在不同的节点上，因为是在本地的IDE上跑的web项目，所以直接在同一个节点上用两个不同的端口号来部署了。）

nginx_master 192.168.146.154　 [nginx + keepalived]
nginx_backup 192.168.146.156　 [nginx + keepalived]
website 192.168.33.239 [web1 + web2]

拓扑图如下：

nginx_master节点上安装nginx+keepalived作为nginx的主要工作节点，
nginx_backup节点上同样安装nginx+keepalived作为nginx的备用节点，
通过keepalived组成高可用集群，当主节点宕机之后，keepalived自动切换到备用的nginx_backup节点。
其中vip（虚拟ip）是keepalived配置网卡之后产生的，将vip与nginx_master和nginx_master的ip绑定，在主备切换时不改变ip，对客户端做到无感知。

安装主节点的nginx，通过源码编译的方式安装。

下载nginx的源码包到 /opt 目录下。在官网下载，截至2020-3-25日最新的稳定版本是1.16.1
# cd /opt && wget https://nginx.org/download/nginx-1.16.1.tar.gz
将下载的压缩包解压到 /usr/local 目录下
# tar -zxvf nginx-1.16.1.tar.gz -C /usr/local
安装编译所需的依赖。系统自带会有zlib和pcre，但是编译过程中需要zlib-devel和pcre-devel的包。
# yum install -y zlib-devel pcre-devel
设置编译后安装的目录为/usr/local/nginx
# cd /usr/local/nginx-1.16.1 && ./configure --prefix=/usr/local/nginx
进行编译安装
# make && make install
编译过程最后一行有以下报错，不用管。

make[1]: Leaving directory `/usr/local/nginx-1.16'

进入niginx的安装目录的sbin目录中，启动nginx
# cd /usr/local/nginx/sbin && ./nginx
访问 http://192.168.146.154 ，出现nginx的欢迎页面，则表示nginx安装成功。

设置主节点nginx的配置文件，让它变成两台web的反向代理。

备份配置文件
# cp /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf.bak
修改配置文件
# vi /usr/local/nginx/conf/nginx.conf

http{
   ...
   upstream myapp1 { # 这里是配置反向代理的服务器组。有多少台需要做反向代理的就写多少台
        server 192.168.33.239:8001;
        server 192.168.33.239:8002;
   }
   server {
       listen   9002;  # nginx监听的本机的端口
       location = / {
           proxy_pass http://myapp1/;  # 反向代理上面配置的服务器组
       }
   }
   ...
}

可以重新加载nginx的配置文件也可以重新启动。这里选择重新加载配置文件
# /usr/local/nginx/sbin/nginx -s reload
这个时候通过访问 http://192.168.146.154:9002 即可访问到后端的web服务器。已经实现了负载均衡。

安装和配置主节点上的keepalived。

通过yum的方式安装keepalived。
# yum install -y keepalived
安装完之后，可以通过命令查看keepalived的安装位置。
# rpm -ql $(rpm -qa keepalived)

/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/lib/systemd/system/keepalived.service
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.3.5
...

可以看到keepalived的配置文件位于/etc/keepalived/keepalived.conf, 通过systemctl命令启动的配置文件在/etc/sysconfig/keepalived。一般我们都会想知道它的日志输出位置，这里可以通过修改/etc/sysconfig/keepalived 文件来指定日志输出位置。

配置keepalived的日志输出，修改前先备份
# vi /etc/sysconfig/keepalived

KEEPALIVED_OPTIONS="-D -S 0"  # '-S 0' 指定输出到rsyslog的local0.*

接下去修改rsyslog的配置，修改前先备份, 将local0.*指定到 /var/log/keepalived.conf
# cp /etc/rsyslog.conf /etc/rsyslog.conf.bak && vi /etc/rsyslog.conf

...
# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
local0.*                                               /var/log/keepalived.log
...

然后接下去重启rsyslog。
# systemctl restart rsyslog
这时候已经指定好keepalived的日志输出路径了。接下去是要修改keepalived的配置，让它绑定虚拟ip，并且检测主nginx
# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak && vi /etc/keepalived/keepalived.conf

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id nginx_master # 最主要的配置，绑定nginx主节点的ip地址。 nginx_master 要在/etc/hosts里面做IP映射。
}

vrrp_script chk_http_port {
        script "/usr/local/nginx/check_nginx_pid.sh" # 检测nginx是否宕机的脚本。
        interval 2 # 检测的间隔，2s
        weight 2 
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33       #绑定网卡，通过这个网卡接口来产生虚拟ip。 通过'ip addr'命令查看。
    virtual_router_id 51  #路由id，主和备两个的id要一样
    priority 100           # 权重，0-100， 权重值越大越优先。 一般配置是主节点的比备用的大
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.146.50     #虚拟ip地址。主要网段，前三个网段跟nginx主节点一样，最后一个网段不一样。
    }
}

编写主节点上的检测脚本
# vi /usr/local/nginx/check_nginx_pid.sh

#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
    /usr/local/nginx/sbin/nginx                #重启nginx
    if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then    #nginx重启失败
        exit 1
    else
        exit 0
    fi
else
    exit 0
fi

开启keepalived. 开启之后就可以通过看/var/log/keepalived.log查看日志
systemctl start keepalived

Jan  9 09:35:15 node2 Keepalived[30887]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Jan  9 09:35:15 node2 Keepalived[30887]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 09:35:15 node2 Keepalived[30888]: Starting Healthcheck child process, pid=30889
Jan  9 09:35:15 node2 Keepalived[30888]: Starting VRRP child process, pid=30890
Jan  9 09:35:15 node2 Keepalived_healthcheckers[30889]: Initializing ipvs
Jan  9 09:35:15 node2 Keepalived_healthcheckers[30889]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: Registering Kernel netlink reflector
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: Registering Kernel netlink command channel
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: Registering gratuitous ARP shared channel
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: Using LinkWatch kernel netlink reflector...
Jan  9 09:35:15 node2 Keepalived_vrrp[30890]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jan  9 09:35:16 node2 Keepalived_vrrp[30890]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan  9 09:35:17 node2 Keepalived_vrrp[30890]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan  9 09:35:17 node2 Keepalived_vrrp[30890]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan  9 09:35:17 node2 Keepalived_vrrp[30890]: Sending gratuitous ARP on ens33 for 192.168.146.50
Jan  9 09:35:17 node2 Keepalived_vrrp[30890]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.146.50

这时候还可以通过查看网卡，是否有我们配置的虚拟ip。
# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:da:56:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.146.154/24 brd 192.168.146.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.146.50/32 scope global ens33   # 这个ip就是我们配置的虚拟ip，证明keepalived配置成功了。
       valid_lft forever preferred_lft forever
    inet6 fe80::3c49:28a5:b36c:2272/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

配置备份节点上的nginx和keepalived

安装步骤跟上面的一样。主要是配置文件的区别。

nginx的配置文件跟主节点的一样。
keepalived的配置文件如下。指定日志输出位置的方法跟主机点的一样，参考上面的方法。

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id nginx_backup # 与主节点的区别，绑定nginx备份节点的ip地址。 nginx_backup 要在/etc/hosts里面做IP映射。
}

vrrp_script chk_http_port {
        script "/usr/local/nginx/check_nginx_pid.sh" # 检测nginx是否宕机的脚本。
        interval 2 # 检测的间隔，2s
        weight 2 
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33       #绑定网卡，通过这个网卡接口来产生虚拟ip。 通过'ip addr'命令查看。
    virtual_router_id 51  #路由id，主和备两个的id要一样
    priority 90           # 权重，0-100， 权重值越大越优先。 一般配置是主节点的比备用的大
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.146.50     #虚拟ip地址。主要网段，前三个网段跟nginx主节点一样，最后一个网段不一样。
    }
}

编写备份节点上的检测脚本
# vi /usr/local/nginx/check_nginx_pid.sh

#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
    /usr/local/nginx/sbin/nginx                #重启nginx
    if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then    #nginx重启失败
        exit 1
    else
        exit 0
    fi
else
    exit 0
fi

开启备份节点的nginx 和 keepalived
# /usr/local/nginx/sbin/nginx && systemctl start keepalived

检测效果

访问 http://192.168.146.50:9002
停止主节点的 nginx + keepalived，然后再访问 http://192.168.146.50:9002。

如果以上访问都可以正确返回，那么就确认高可用集群搭建成功。

posted @ 2020-03-26 16:40 xstar-website 阅读(361) 评论(0) 编辑收藏举报

刷新页面返回顶部