keepalived的状态不断切换的问题解决

转载自：https://blog.csdn.net/weixin_43515220/article/details/104959814

=================

 

笔者在搭建nginx+keepalived架构的过程中，发现存在keepalived的vip不断迁移的过程：master down了之后，vip一会迁移到backup上，一会迁移到master上，或者在backup上一会存在一会消失。

就这样的情况笔者做了下面排查。

先介绍下环境：

master和backup都为CentOS Linux release 7.7.1908 (Core)，均不开启防火墙

master IP： 192.168.218.131

backup IP：192.168.218.132

VIP：192.168.218.140

通过keepalived执行脚本检测Nginx的情况，若Nginx挂掉了，keepalived也会停止

Nginx的配置其实已经没有问题，关键是keepalived

下面是master的keepalived配置

复制代码
! Configuration File for keepalived

global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}

vrrp_script check_nginx {
script "/usr/local/nginx/sbin/check_nginx.sh"
}

vrrp_instance VI_1 {
state MASTER
interface ens33 # 修改网卡
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.218.140/24 # vip for k8s master
}
track_script {
check_nginx
}
}
复制代码

 


下面是backup的keepalived配置

复制代码
! Configuration File for keepalived

global_defs {
# 接收邮件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}

vrrp_script check_nginx {
script "/usr/local/nginx/sbin/check_nginx.sh"
}

vrrp_instance VI_1 {
state BACKUP # BACKUP
interface ens33
virtual_router_id 51
priority 90 # 优先级小于master
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.218.140/24 # vip for k8s master
}
track_script {
check_nginx
}
}
复制代码

 

下面是keepalived中的执行脚本check+nginx.sh，已加执行权限

count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi

 

排查

在master上手动pkill nginx之后，检查nginx状态，nginx已经停止

但是再检查keepalived的时候，keepalived还是正在运行

笔者先是在backup上查看系统日志

vim /var/log/messages

复制代码
Mar 18 23:53:46 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.218.140
Mar 18 23:53:46 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:46 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:46 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:46 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:48 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90
Mar 18 23:53:48 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Entering BACKUP STATE
Mar 18 23:53:48 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) removing protocol VIPs.
Mar 18 23:53:51 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.218.140
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:52 nginx02 Keepalived_vrrp[1365]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 18 23:53:53 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90
Mar 18 23:53:53 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Entering BACKUP STATE
Mar 18 23:53:53 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) removing protocol VIPs.
Mar 18 23:53:57 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar 18 23:53:58 nginx02 Keepalived_vrrp[1365]: VRRP_Instance(VI_1) Entering MASTER STATE
复制代码

在日志中可以看到，backup一会切到BACKUP状态，一会切到MASTER状态，根据提示，切换的原因是存在一个优先级为100的主机，比backup的优先级90要高

在这里笔者觉得很奇怪，优先级为100的那不就是master吗？

master的keepalived优先级没有因为Nginx挂掉而发生改变

在网上查阅了一些优秀博客之后，得到以下信息：

keepalived的vrrp_script执行后异常退出，可通过weight -int 的形式降低keepalived的优先级

于是针对笔者的实际情况，做出以下修改:

# 修改keepalived的vrrp_script：/usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
systemctl stop keepalived
exit 1 # 当nginx挂掉后，返回退出状态码1，以使其异常退出
fi

 

修改master和backup的keepalived配置：

# 两台主机都在vrrp_script中修改
vrrp_script check_nginx {
script "/usr/local/nginx/sbin/check_nginx.sh"
weight -20 # 脚本异常退出后，使keepalived的优先级-20
interval 2 # 脚本每2秒执行一次
}

 

将keepalived和Nginx的状态复原后，再次在master上pkill nginx

发现vip已迁移到backup上，且不发生原来的不断迁移的情况

下面查看backup的系统日志/var/log/messages

复制代码
Mar 19 01:05:25 nginx02 Keepalived_vrrp[1417]: /usr/local/nginx/sbin/check_nginx.sh exited with status 127
Mar 19 01:05:26 nginx02 Keepalived_vrrp[1417]: /usr/local/nginx/sbin/check_nginx.sh exited with status 127
Mar 19 01:05:27 nginx02 Keepalived_vrrp[1417]: /usr/local/nginx/sbin/check_nginx.sh exited with status 127
Mar 19 01:05:28 nginx02 Keepalived_vrrp[1417]: /usr/local/nginx/sbin/check_nginx.sh exited with status 127
Mar 19 01:05:29 nginx02 Keepalived_vrrp[1417]: /usr/local/nginx/sbin/check_nginx.sh exited with status 127
Mar 19 01:05:30 nginx02 Keepalived_vrrp[1417]: VRRP_Script(check_nginx) succeeded

...省略部分信息...

Mar 19 01:06:40 nginx02 Keepalived_vrrp[1417]: VRRP_Instance(VI_1) Transition to MASTER STATE
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: VRRP_Instance(VI_1) Entering MASTER STATE
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: VRRP_Instance(VI_1) setting protocol VIPs.
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.218.140
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: Sending gratuitous ARP on ens33 for 192.168.218.140
Mar 19 01:06:41 nginx02 Keepalived_vrrp[1417]: Sending gratuitous ARP on ens33 for 192.168.218.140
复制代码

 

posted @   larybird  阅读(730)  评论(0)  编辑  收藏  举报
相关博文：
· kubeadm方式部署k8s集群（三主两从）
· 重置k8s集群，主master节点备份
· Nginx+keepalived 双机热备（主从模式）
· keepalived的简单使用
· HA-Keepalived-01-Keepalived
阅读排行：
· 分享4款.NET开源、免费、实用的商城系统
· 全程不用写代码，我用AI程序员写了一个飞机大战
· MongoDB 8.0这个新功能碉堡了，比商业数据库还牛
· 白话解读 Dapr 1.15：你的「微服务管家」又秀新绝活了
· 记一次.NET内存居高不下排查解决与启示
历史上的今天：
2022-02-09 python测试开发django-67.templates模板变量取值
2022-02-09 解决Django生产环境无法加载静态文件404问题的解决
2022-02-09 uwsgi命令
2022-02-09 ”ModuleNotFoundError:No module named _ctypes“ 引发的一系列问题解决办法
2022-02-09 django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17).
2022-02-09 解决Python3 No module named '_sqlite3'错误
2022-02-09 Linux下切换Python版本的几种方法
点击右上角即可分享
微信分享提示