keepalived+lvs 部署
环境配置：
| node | service | VIP |
|---|---|---|
| node2010(192.168.20.10) | keepalived(master)+lvs | 192.168.20.22 |
| node2011(192.168.20.11) | keepalived(backup)+lvs | |
| node2012(192.168.20.12) | httpd(RS) |
lvs相关内容请翻看另外内容
安装
$ yum install -y curl gcc openssl-devel libnl3-devel net-snmp-devel
$ tar xf keepalived-2.0.19.tar.gz
$ cd keepalived-2.0.19
$ ./configure --prefix=/usr/local/keepalived
$ make -j 4 && make install
$ tree -L 3 /usr/local/keepalived
/usr/local/keepalived
├── bin
│ └── genhash
├── etc
│ ├── keepalived
│ │ ├── keepalived.conf #主配置文件
│ │ └── samples
│ └── sysconfig
│ └── keepalived
├── sbin
│ └── keepalived
└── share
├── doc
│ └── keepalived
├── man
│ ├── man1
│ ├── man5
│ └── man8
└── snmp
└── mibs
#添加systemd
$ vim /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived -f /usr/local/keepalived/etc/keepalived/keepalived.conf
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
$ systemctl daemon-reload && systemctl enable keepalived.service
配置keepalived.conf
$ cat /usr/local/keepalived/etc/keepalived/keepalived.conf
!Configuration File for keepalived
global_defs { //全局定义，邮件配置。
# notification_email {
# localhost@root.com
#}
# notification_email_from root
# smtp_server 127.0.0.1
# smtp_connect_timeout 30
router_id LVS_DEVEL //标识虚拟路由ID,主从得不同
vrrp_mcast_group4 224.0.10.10 //组播地址，用来发送VRRP报文
# enable_script_security
# script_user root //脚本执行者，建议使用非root来执行
}
#vrrp_script chk_schedown { //自检可以通过第三方监控程序zabbix,prometheus
# script "[ -e /etc/keepalived/down ] && exit 1 || exit 0"
# interval 1
# weight -11
# fall 2
# rise 1
#}
#vrrp_script chk_nginx { //此处检测nginx只需在主上有即可，如果从上也有话，就会导致主上的nginx，down的一瞬间，又起来了，优先级混乱
# script "`killall -0 nginx` && exit 0 || exit 1"
# interval 1
# weight -11
# fall 2
# rise 1
#}
vrrp_instance VI_1 {
state MASTER //主，BACKUP为备
interface ens192 //虚拟IP绑定的网卡
virtual_router_id 10 //虚拟路由ID组，这个主从得相同
priority 100 //优先级，高的为MASTER，BACKUP需比此小
advert_int 1 //一秒一次VRRP报文
authentication { //VRRP报文简单加密
auth_type PASS
auth_pass 12345678
}
virtual_ipaddress { //VIP地址，也可写多个
192.168.20.22
}
# track_script { //调用脚本，做健康状态检测
# chk_schedown
# #chk_nginx
# }
#notify_master "/etc/keepalived/notify.sh master"
#notify_backup "/etc/keepalived/notify.sh backup"
#notify_fault "/etc/keepalived/notify.sh fault"
notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master" #状态发生变化时执行的脚本
}
#keepalived与lvs结合时的配置
virtual_server fwmark 2 { #IP port(VIP)、fwmark int(防火墙标记)、group string(虚拟服务器组)
delay_loop 6 //延迟定时器用于服务轮询
lb_algo wrr //调度算法
lb_kind DR //lvs模式
# nat_mask 255.255.255.0 //新版本keepavlied不支持以下两种配置
# persistence_timeout 0
protocol TCP
real_server 192.168.20.3 { //后端RS
weight 1 //权重
TCP_CHECK { //RS健康检测，检测方式有：HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK
connect_timeout 10
#nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.20.12 {
weight 3
TCP_CHECK {
connect_timeout 10
#nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
脚本(主要作用是添加iptables mark 标记)
mark标记是为了使用lvs能够映射多个端口至RS
$ cat /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash
VIP=192.168.20.22
case $1 in
master)
iptables -t mangle -L > /tmp/iptables
mangle_rule=`grep "multiport dports http,https MARK set 0x2" /tmp/iptables | wc -l`
if [ $mangle_rule -ne 1 ];then
iptables -t mangle -F
iptables -t mangle -A PREROUTING -d $VIP -p tcp -m multiport --dports 80,443 -j MARK --set-mark 2
[ $? -ne 0 ] && echo "iptables write false!!!";systemctl stop keepalived.service
fi
;;
*)
echo "Please write iptables to add mangle rules. "
esac
启动服务
$ systemctl start keepalived.service //MASTER 机器查看
$ ip a
...
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:7c:ab:08 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.10/24 brd 192.168.20.255 scope global ens192
valid_lft forever preferred_lft forever
inet 192.168.20.22/32 scope global ens192 #VIP已经加入
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe7c:ab08/64 scope link
valid_lft forever preferred_lft forever
$ ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 2 wrr
-> 192.168.20.12:0 Route 3 0 0 #ipvs 规则已经加入，这里192.168.20.3未安装httpd,所以TCP检测失败次数超过设置后将规则移除
$ iptables -t mangle -L #防火墙标记添加，这样http,https端口都可以转发至RS上
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere node2010 multiport dports http,https MARK set 0x2
...
#从内网其它机器访问VIP
$~]# curl http://192.168.20.22
12
抓包查看VRRP通信过程
$ tcpdump -i ens192 host 224.0.10.10
11:56:25.574398 IP node2010 > 224.0.10.10: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
11:56:26.574467 IP node2010 > 224.0.10.10: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
...
关闭MASTER，并查看VIP是否漂移至BACKUP
node2010$ systemctl stop keepalived
node2011$ less /var/log/message
Jan 16 17:40:43 node2011 Keepalived_vrrp[25872]: (VI_1) Backup received priority 0 advertisement #Backup主机接收到优先级为0的VRRP
Jan 16 17:40:44 node2011 Keepalived_vrrp[25872]: (VI_1) Entering MASTER STATE #0 < 90 ,于是BACKUP转换成MASTER
node2011$ ip a
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:f3:c4:42 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.11/24 brd 192.168.20.255 scope global ens192
valid_lft forever preferred_lft forever
inet 192.168.20.22/32 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fef3:c442/64 scope link
valid_lft forever preferred_lft forever
node2011$ ipvsadm -ln #切换成功，VIP已经漂移
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 2 wrr
-> 192.168.20.12:0 Route 3 0 0
监控
最有效的监控方式一、使用自带的邮件发送功能，二、监控日志，状态转换会写日志也会触发脚本。脚本和第三方监控程序还是会有时间间隔。
建议使用服务自带的邮件功能即可，实时性最好。同时也可以用prometheus对进程状态进行监控。双管其下。
结语
在操作过程中，可以通过
arp -a来查看VIP与对应MAC地址是否相匹配。一但出现不匹配那就是你配置问题了。
参考资料：
