LVS(dr)+keepalived
LVS可以实现负载均衡,但是不能进行健康检查,如果有一个rs出现故障,LVS任然会把请求转发给故障的rs服务器,这样就会导致请求失败。
keepalived可以进行健康检查,同时能够实现LVS的高可用性,解决LVS的单点故障问题,其实,keepalived就是为LVS而生的。
实验环境:四台机器
192.168.75.61: LVS1 BACKUP
192.168.75.63: LVS2 MASTER
192.168.75.64:realserver1
192.168.75.65:realserver2
LVS的两个节点安装ipvsadm+keepalived
yum install -y ipvsadm keepalived(本文我是用编译安装的keepalived)
realserver的节点安装nginx
yum install -y nginx
配置脚本:
realserver两台节点的配置脚本:
[root@VM-75-64 ~]# cat lvs_dr_rs.sh
#!/bin/bash
vip=192.168.75.55
set -x
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
由于之前已经配置过dr模式的lvs,lo:0网卡已经存在了,因此首先要手动删掉对应的网卡,操作步骤为:
#ifconfig lo:0 down
在两台rs上分别执行该脚本。
LVS两台节点配置:ipvsadm不用配置了,直接在keepalived的配置文件里配置,所以说keepalived是为lvs而生的。
两台节点的keepalived配置文件:
MASTER:192.168.76.63
[root@VM-75-63 keepalived]# pwd
/usr/local/etc/keepalived
[root@VM-75-63 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 75_63
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.75.55
}
}
virtual_server 192.168.75.55 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.75.64 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 10
}
}
real_server 192.168.75.65 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
BACKUP:192.168.75.61
[root@VM-75-61 keepalived]# pwd
/usr/local/etc/keepalived
[root@VM-75-61 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 75_61
vrrp_skip_check_adv_addr
# vrrp_strict #这里要注释掉,否则虚拟出的IP是ping不通的
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.75.55
}
}
virtual_server 192.168.75.55 80 { #此处等同于:/sbin/ipvsadm -A -t 192.168.75.55:80 -s wrr
delay_loop 6 #服务轮询的时间间隔,每隔6秒检测后端服务
lb_algo rr #lvs的调度算法
lb_kind DR #lvs集群模式
persistence_timeout 360 #持久化超时时间,默认是6分钟,客户端在持久化超时时间内的连接,会持续的连接到同一台后端服务器
protocol TCP #指定协议
real_server 192.168.75.64 80 { #等同于/sbin/ipvsadm -a -t 192.168.75.55:80 -r 192.168.75.64:80 -g -w 1
weight 1 #权重
TCP_CHECK {
connect_port 80
connect_timeout 3 #检测的连接超时时间
nb_get_retry 3 #重试次数,这里是HTTP_GET检测方式的配置字段,TCP_CHECK一般可以去掉
delay_before_retry 10 #重试之前延迟多少秒,默认1s
}
}
real_server 192.168.75.65 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
keepalived的配置文件处理好之后,开启keepalived进程:
注意,在开启keepalived的时候,先MASTER,再BACKUP!
[root@VM-75-63 init.d]# /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
[root@VM-75-63 init.d]# ps -ef |grep keep
root 27612 1 0 04:18 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 27613 27612 0 04:18 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 27614 27612 0 04:18 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 27619 8470 0 04:18 pts/0 00:00:00 grep keep
进程起来了,看下状态:
[root@VM-75-63 init.d]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:d1:74:a7 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.63/24 brd 192.168.75.255 scope global eth0
inet 192.168.75.55/32 scope global eth0 #这一行,就是虚拟IP
inet6 fe80::20c:29ff:fed1:74a7/64 scope link
valid_lft forever preferred_lft forever
同样开启BCKUP的keepalived:
[root@VM-75-61 sysconfig]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:b3:00:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.61/24 brd 192.168.75.255 scope global eth0
inet6 fe80::250:56ff:feb3:1/64 scope link
valid_lft forever preferred_lft forever
此时并没有VIP,因为VIP在MASTER节点上!
ok,在MASTER节点查看ipvsadm的结果:
[root@VM-75-63 init.d]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.75.55:http rr
-> 192.168.75.64:http Route 1 0 0
-> 192.168.75.65:http Route 1 0 0
这里我们就不需要手动去配置虚拟主机和真实主机,而是通过keepalived的配置文件去实现的,所以说二者的联系很密切。
虽然vip不在backup节点上,但是ipvsadm的配置,他是存在的:
[root@VM-75-61 sysconfig]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.75.55:http rr
-> 192.168.75.64:http Route 1 0 0
-> 192.168.75.65:http Route 1 0 0
现在,测试在页面访问:http://192.168.75.55
发现访问不不通,连接超时,但是当我们用同一网段的linux虚拟机访问时:
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.64
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.64
发现能够拿到结果,并且轮询方式也是rr,那么现在思考,为什么用局域网内的办公电脑没法访问呢???
办公电脑跟后端realserver在不同网段,不同网段通信就需要添加路由信息,好的,查看realserver的路由信息:
[root@VM-75-64 ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.75.55 * 255.255.255.255 UH 0 0 0 lo
192.168.75.0 * 255.255.255.0 U 0 0 0 eth1
link-local * 255.255.0.0 U 1002 0 0 eth1
default 192.168.75.55 0.0.0.0 UG 0 0 0 eth1
default 192.168.75.1 0.0.0.0 UG 0 0 0 eth1
我们看到192.168.75.55 * 255.255.255.255 UH 0 0 0 lo 这条是到75的路由信息,且后面的网卡设备是lo,但是往下发现,本地还有个默认到75.55的网关,通过上面学习的结果知道,lvs的DR模式下,是不允许后端的realserver配置默认IP为VIP的网关的,好的,我们尝试删除该条网关:
[root@VM-75-64 ~]# route del default gw 192.168.75.55
[root@VM-75-64 ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.75.55 * 255.255.255.255 UH 0 0 0 lo
192.168.75.0 * 255.255.255.0 U 0 0 0 eth1
link-local * 255.255.0.0 U 1002 0 0 eth1
default 192.168.75.1 0.0.0.0 UG 0 0 0 eth1
好的,再次尝试用页面访问:
你看,访问到了,思考下原因:
因为DR模式是realserver直接把响应包反馈给客户端的,也就是这里的web主机,我电脑的IP,76.147,这里看来二者没法通信,虽然加上了路由信息,但默认网关是走的vip,也就是这里的75.55,但是实际上返回包是通过realserver的lo的IP,也就是vip,传给eth0,也就是75.64,再返回给web,所以这里需要网关是75.1,也就是真实的默认网关,不然76.147和75.64是没法通信的。
测试1:
手动关闭75.64的nginx进程:
[root@VM-75-64 ~]# service nginx stop
Stopping nginx: [ OK ]
测试访问情况:
[root@Vm-75-60 ~]# curl http://192.168.75.55
curl: (7) couldn't connect to host
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
curl: (7) couldn't connect to host
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
curl: (7) couldn't connect to host
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
[root@Vm-75-60 ~]# curl http://192.168.75.55
75.65
期初你会发现,当轮询到75.64上时,访问是会失败的,75.65节点正常,但是大约10秒钟之后,所有的访问请求都会被分配到75.65节点上,这是因为keepalived检测到后端有节点响应失败,所以把所有的请求都分配到正常的节点上了!
当75.64的nginx再次开起来之后,realserver立马加入集群,处理请求。
测试2:
keepalived的HA特性,我们尝试关闭MASTER的keepalived进程,看VIP是否能够正常的票到BACKUP节点上:
[root@VM-75-63 keepalived]# ps -ef |grep keep
root 27612 1 0 04:18 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 27613 27612 0 04:18 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 27614 27612 0 04:18 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 27722 8470 0 06:40 pts/0 00:00:00 grep keep
[root@VM-75-63 keepalived]#
[root@VM-75-63 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:d1:74:a7 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.63/24 brd 192.168.75.255 scope global eth0
inet 192.168.75.55/32 scope global eth0 #VIP在MASTER上
inet6 fe80::20c:29ff:fed1:74a7/64 scope link
valid_lft forever preferred_lft forever
[root@VM-75-63 keepalived]# pkill keepalived
[root@VM-75-63 keepalived]# ps -ef |grep keep
root 27726 8470 0 06:41 pts/0 00:00:00 grep keep
[root@VM-75-63 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:d1:74:a7 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.63/24 brd 192.168.75.255 scope global eth0
inet6 fe80::20c:29ff:fed1:74a7/64 scope link #VIP不见了
valid_lft forever preferred_lft forever
MASTER上的VIP不见了,我们看下BACKUP的情况:
[root@VM-75-61 sysconfig]# ps -fe | grep keep
root 16821 1 0 09:49 ? 00:00:00 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 16822 16821 0 09:49 ? 00:00:01 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 16823 16821 0 09:49 ? 00:00:01 /usr/local/sbin/keepalived -f /usr/local/etc/keepalived/keepalived.conf
root 17076 31656 0 15:01 pts/0 00:00:00 grep keep
[root@VM-75-61 sysconfig]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:b3:00:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.61/24 brd 192.168.75.255 scope global eth0
inet 192.168.75.55/32 scope global eth0
inet6 fe80::250:56ff:feb3:1/64 scope link
valid_lft forever preferred_lft forever
是不是,来到75.61上了!并且服务依旧是可用的,那么这里就实现了双机热备的高可用架构。
注意:实验初期,我在测试VIP漂移的时候,发现MASTER的keepalived杀掉之后,VIP并不会漂移,依旧在MASTER上,更奇怪的是,BACKUP上也出现了VIP,二者同时存在,这是为什么呢?而且通过抓包我们发现:
[root@VM-75-61 sysconfig]# tcpdump -i eth0 -p vrrp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:05:11.078778 IP 192.168.75.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:05:12.079844 IP 192.168.75.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:05:13.080878 IP 192.168.75.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:05:14.081893 IP 192.168.75.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
vrrp广播确实由75.61也就是BACKUP发送的,也就说是由75.61提供服务的,并且服务依旧可用!
当把master的keepalived开起来之后,backup的VIP就没了,通过抓包发下:
[root@VM-75-61 sysconfig]# tcpdump -i eth0 -p vrrp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:05:44.108400 IP 192.168.75.63 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 150, authtype simple, intvl 1s, length 20
15:05:45.109462 IP 192.168.75.63 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 150, authtype simple, intvl 1s, length 20
15:05:46.110508 IP 192.168.75.63 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 150, authtype simple, intvl 1s, length 20
又由master提供服务了,所以这里的情况是:VIP仅仅看起来没有转移,实际上高可用功能是能够正常提供的,那么为什么呢?
通过网上提供的思路发现,我在杀master端的keepalived进程,使用了 -9 参数,也就是强制中断,并不是正常的关闭进程,当我尝试不加参数‘-9’的时候,VIP顺利的飘走了,服务依旧可用,那么,这里在测试的时候要注意了!
不能使用 ‘kill -9’
好的,关于lvs的DR模式的高可用大致就说到这里,后面有发现会继续补充。
LVS状态能够通过zabbix自动以监控进行监控并画图,监控数据主要从下面几个指令得到:
[root@VM-75-63 bin]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP C0A84B37:0050 rr
-> C0A84B41:0050 Route 1 0 0
-> C0A84B40:0050 Route 1 0 0
这里看起来不太友好,换个方式:
[root@VM-75-63 bin]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.75.55:http rr
-> 192.168.75.64:http Route 1 0 0
-> 192.168.75.65:http Route 1 0 0
这里是展示连接数的。
[root@VM-75-63 bin]# cat /proc/net/ip_vs_stats
Total Incoming Outgoing Incoming Outgoing
Conns Packets Packets Bytes Bytes
542 58DB 3E99 8C1065 344D49
Conns/s Pkts/s Pkts/s Bytes/s Bytes/s
1 7 0 214 0
上面的指令是展示当前往来的数据量和输出带宽的。
自定义监控项的具体细节就不阐述了。
以上,共勉!