开发运维日记(二) keepalived实现VIP和nginx可用性检查
上篇中已经搭建了Nginx集群，但还缺少高可用的方案。开源的组件keepalived可以方便发送心跳监控Nginx并实现VIP，所有最终选择keepalived实现高可用。
1、安装环境：centos7 、keepalived-1.3.5
2、主机：192.168.30.7 从机：192.168.30.7 VIP：192.168.30.77
由于主从服务器配置一样，只演示一台机器的配置。
1、安装keepalived（使用源码安装）
==> 安装库依赖
yum install -y libnl* yum install -y libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh
==> 编译源码并安装（将keepalived-1.3.5.tar.gz下载至当前目录）
tar xvf keepalived-1.3.5.tar.gz cd keepalived-1.3.5 ./configure --prefix=/usr/local/keepalived make && make install
此时会在对应的路径生产以下的文件
（1）/usr/local/etc/keepalived/keepalived.conf
（2）/usr/local/etc/sysconfig/keepalived
（3）/usr/local/sbin/keepalived
==> 初始化配置，设置为系统服务
（1）keepalived启动脚本变量引用文件，copy至默认文件路径是/etc/sysconfig/，也可以不做软链接，直接修改启动脚本中文件路径即可（安装目录下）
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
（2）将keepalived主程序加入到环境变量（安装目录下）
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
（3）keepalived启动脚本（源码目录下），放到/etc/init.d/目录下就可以使用service命令便捷调用
cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/keepalived
（4）将配置文件放到默认路径下，此时启动keepalive服务默认读取该配置
mkdir -p /etc/keepalived cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
（5）将keepalived设置为系统服务 开机启动
设置为系统服务开机启动
chkconfig --add keepalived chkconfig keepalived on
查看开机启动服务
chkconfig –-list
启动服务|关闭|重启命令
service keepalived start|stop|restart
主从配置一样，这里不赘述
2、配置VIP和nginx可用性检查
==> 在/etc/keepalived创建 check_nginx.sh
#!/bin/bash
A = `ps aux | grep nginx | grep -v grep | wc -l`
r = 2
if test $[A] -le $[r]
then
/usr/local/nginx/sbin/nginx
sleep 2
re = `ps axu | grep nginx | grep -v grep |wc -l`
if test $[re] -le $[r]
then
service keepalived stop
fi
fi
添加执行权限
chmod 755 /etc/keepalived/check_nginx.sh
==> 修改主节点keepalived.conf（主）
global_defs {
router_id lb-ecg
enable_script_security
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 5
weight -2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 240
mcast_src_ip 192.168.30.7
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass nginx
}
virtual_ipaddress {
192.168.30.77/24
}
track_interface {
ens33
}
track_script {
chk_nginx
}
}
==> 修改从节点keepalived.conf（从）
global_defs {
router_id lb-ecg
enable_script_security
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 5
weight -2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 240
mcast_src_ip 192.168.30.8
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass nginx
}
virtual_ipaddress {
192.168.30.77/24
}
track_interface {
ens33
}
track_script {
chk_nginx
}
}
==> 启动keepalived
service keepalived start
注意点：
（1）需要打开 enable_script_security 否则会 【security violation - scripts are being executed but script_security not enabled.】
（2）check_nginx.sh需要赋予755权限否则会 【keepalived unsafe permissions found for script】
3、VIP漂移检查 和 nginx检测
当keepalived、nginx配置完成并启动机器192.168.30.7/8
==> 192.168.30.7机器：此时VIP(192.168.30.77)绑定在192.168.30.7的ens33的网卡上
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.30.77/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@localhost keepalived]#
==> 192.168.30.8机器：未绑定VIP(192.168.30.77)
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
[root@localhost keepalived]#
==> 验证性测试：
（1）kill掉192.168.30.7的keepalived，VIP(192.168.30.77)将漂移到192.168.30.8上。重新启动192.168.30.7的keepalived，VIP(192.168.30.77)将漂回192.168.30.7。
结果如下： 192.168.30.7机器
[root@localhost keepalived]# ps aux | grep keepalived root 49997 0.0 0.0 48088 1048 ? Ss 00:39 0:00 /usr/local/keepalived/sbin/keepalived -D root 49998 0.0 0.0 48088 1792 ? S 00:39 0:00 /usr/local/keepalived/sbin/keepalived -D root 49999 0.0 0.0 48088 1392 ? S 00:39 0:00 /usr/local/keepalived/sbin/keepalived -D root 92683 0.0 0.0 48088 628 ? S 01:09 0:00 /usr/local/keepalived/sbin/keepalived -D root 92684 0.0 0.0 115304 1496 ? S 01:09 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 92749 0.0 0.0 112712 972 pts/2 R+ 01:09 0:00 grep --color=auto keepalived [root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.30.77/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost keepalived]# [root@localhost keepalived]# kill -9 49997 [root@localhost keepalived]# ps aux | grep keepalived root 93225 0.0 0.0 112712 968 pts/2 R+ 01:10 0:00 grep --color=auto keepalived [root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost keepalived]# [root@localhost keepalived]# service keepalived start Starting keepalived (via systemctl): ^C [root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.30.77/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost keepalived]#
192.168.30.8机器
。。。此时此时192.168.30.7的keepalived已经被kill掉
[root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.30.77/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever
。。。。 此时192.168.30.7的keepalived重启完毕
[root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever [root@localhost keepalived]#
（2）kill掉192.168.30.7的Nginx, 由于keepalived定时脚本 Nginx进程将自动恢复
[root@localhost keepalived]# ps aux | grep nginx root 113806 0.0 0.0 115304 1496 ? S 01:25 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 113871 0.0 0.0 112712 964 pts/2 R+ 01:25 0:00 grep --color=auto nginx root 127705 0.0 0.0 20568 672 ? Ss 00:04 0:00 nginx: master process /usr/local/nginx/sbin/nginx nobody 127707 0.0 0.0 20960 1636 ? S 00:04 0:00 nginx: worker process [root@localhost keepalived]# /usr/local/nginx/sbin/nginx -s stop [root@localhost keepalived]# ps aux | grep nginx root 114421 0.0 0.0 115304 1496 ? S 01:25 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 114500 0.0 0.0 115304 556 ? S 01:25 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 114502 0.0 0.0 112712 940 ? S 01:25 0:00 grep nginx root 114506 0.0 0.0 112712 964 pts/2 R+ 01:25 0:00 grep --color=auto nginx
。。。等待5秒之后 nginx进程重新启动
[root@localhost keepalived]# ps aux | grep nginx root 114546 0.0 0.0 20568 676 ? Ss 01:25 0:00 nginx: master process /usr/local/nginx/sbin/nginx nobody 114548 0.0 0.0 20960 1388 ? S 01:25 0:00 nginx: worker process root 114775 0.0 0.0 115304 1500 ? S 01:26 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 114783 0.0 0.0 20564 1640 ? S 01:26 0:00 /usr/local/nginx/sbin/nginx root 114813 0.0 0.0 112712 964 pts/2 R+ 01:26 0:00 grep --color=auto nginx [root@localhost keepalived]#
查看keepalived运行情况也可验证：
[root@localhost keepalived]# service keepalived status ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2020-01-08 01:13:22 CST; 15min ago Process: 96569 ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) CGroup: /system.slice/keepalived.service ├─ 96570 /usr/local/keepalived/sbin/keepalived -D ├─ 96571 /usr/local/keepalived/sbin/keepalived -D ├─ 96572 /usr/local/keepalived/sbin/keepalived -D ├─114546 nginx: master process /usr/local/nginx/sbin/nginx ├─114548 nginx: worker process ├─118238 /usr/local/keepalived/sbin/keepalived -D ├─118239 /bin/bash /etc/keepalived/check_nginx.sh └─118302 sleep 2 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Script(chk_nginx) timed out Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: /etc/keepalived/check_nginx.sh exited due to signal 15 Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Changing effective priority from 100 to 98 Jan 08 01:17:37 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Script(chk_nginx) succeeded Jan 08 01:17:37 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Changing effective priority from 98 to 100 [root@localhost keepalived]#
