keepalived高可用
keepalived介绍
Keepalived是一个基于vrrp协议的高可用方案，vrrp协议的软件实现，原生设计的目的为了高可用ipvs服务。
1. vrrp协议
VRRP是一种容错协议，它通过把几台路由设备联合组成一台虚拟的路由设备，并通过一定的机制来保证当主机的下一跳设备出现故障时，可以及时将业务切换到其它设备，从而保持通讯的连续性和可靠性，
- vrrp协议中常见术语：
- 虚拟路由器：Virtual Router
- 虚拟路由器标识：VRID(0-255)
- 物理路由器：master（主设备），backup（备用设备），priority（优先级）
- VIP：Virtual IP
- VMAC：Virtual MAC（00-00-5e-00-01-VIRD）
- 虚拟路由器工作机制如下：
- 根据优先级的大小挑选Master设备，比较优先级的大小，优先级高者当选为Master。
- 当两台优先级相同的路由器同时竞争Master时，比较接口IP地址大小。接口地址大者当选为Master。
备份路由器随时监听Master的状态。 - 当主路由器正常工作时，它会每隔一段时间（Advertisement_Interval）发送一个VRRP组播报文，以通知组内的备份路由器，主路由器处于正常工作状态。
- 当组内的备份路由器一段时间（Master_Down_Interval）内没有接收到来自主路由器的报文，则将自己转为主路由器。
2. HA Cluster配置
2.1 HA Cluster的配置前提
- 各节点时间必须同步（ntp，chrony）。
- 确保iptables及selinux不会成为阻碍。
- 各节点之间可通过主机名互相通信（对KeepAlived并非必须），简易使用/etc/hosts文件实现。
- 各节点之间的root用户可以基于密钥认证的ssh服务完成相互通信（非必须）；
- 配置keepalived的网卡必须支持并开启多播(multicast)功能。（ip link set dev enoxxxxx multicast on|off）
2.2 HA Cluster的虚拟路由器配置
环境：两台linux主机，配置虚拟路由器组，使用 10.1.
- 同步时间
-
安装ntp软件包
[root@ _8_ ~]# yum -y install ntp -
编辑ntp配置文件，使本机作为ntp时间服务器，注释掉多行以server开头的行，添加server 127.127.0.1
-
重启ntp服务
[root@ _9_ ~]# service ntpd restart -
另一台主机2同步时间
[root@ _9_ ~]# ntpdate 10.1.6.11 1 Nov 18:38:03 ntpdate[46881]: adjust time server 10.1.6.11 offset -0.000035 sec
- keepalived 单主模型(10.1.7.19)
-
安装keepalived
[root@ _14_ ~]# yum -y install keepalived -
主机1修改keepalived配置文件，注释掉Virtual server的内容（这里暂不配置）
[root@ _15_ ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost #管理员邮箱 } notification_email_from keepalived@localhost #发送者keepalived smtp_server 127.0.0.1 #邮箱服务器 smtp_connect_timeout 30 #邮件发送超时时间 router_id node1 #当前路由器物理标识符 vrrp_mcast_group4 224.0.200.158 #多播地址（默认开启），应该与其他组成虚拟路由器的主机一致 } vrrp_instance VI_1 { #配置vrp示例，VI_1，随意定义，需唯一 state MASTER #定义当前节点在此虚拟路由器上的初始状态；只能有一个是MASTER，其余都为BACKUP interface eth0 #绑定为当前虚拟路由器使用的物理接口 virtual_router_id 16 #当前虚拟路由器的唯一标识（0-255） priority 100 #当前主机在此虚拟路由器中的优先级 advert_int 1 #vrrp通告的时间间隔 authentication { auth_type PASS #认证类型，PASS为简单认证，AH为复杂认证，推荐使用PASS auth_pass RrpIoZU7 #认证字符 } virtual_ipaddress { 10.1.7.19/16 dev eth0 #配置的接口虚拟ip } } 注：网卡多播功能开启与关闭：ip link set multicast on|off -
使用scp拷贝给另一台主机2，修改对应参数 。
[root@ _15_ ~]# scp /etc/keepalived/keepalived.conf root@10.1.7.11:/etc/keepalived/keepalived.conf 修改 state MASTER 为 state BACKUP priority 100 为 priority 98 备用节点优先级要比主节点低
- 测试
-
启动主节点，能看到启动为MASTER，添加了10.1.7.19的地址
[root@ _1_ ~]# service keepalived start Starting keepalived: [root@ _1_ ~]# tail /var/log/message Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Opening file '/etc/keepalived/keepalived.conf'. Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Configuration is using : 7552 Bytes Nov 1 20:13:44 localhost kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP) Nov 1 20:13:44 localhost kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes) Nov 1 20:13:44 localhost kernel: IPVS: ipvs loaded. Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Using LinkWatch kernel netlink reflector... Nov 1 20:13:44 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:13:45 localhost Keepalived_healthcheckers[36312]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:13:47 localhost ntpd[2238]: Listen normally on 8 eth0 10.1.7.19 UDP 123 Nov 1 20:13:50 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 [root@ _2_ ~]# ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever -
启动备用节点,可看到由于主节点正常运行，备用节点并未抢占主节点ip
[root@ _3_ ~]# service keepalived start Starting keepalived: [ OK ] [root@ _1_ ~]# tail /var/log/message Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Opening file '/etc/keepalived/keepalived.conf'. Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Configuration is using : 7556 Bytes Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Using LinkWatch kernel netlink reflector... [root@ _4_ ~]# ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever -
关闭主节点keepalived服务后
[root@ _6_ ~]# service keepalived stop Stopping keepalived: [ OK ] 以下为主节点日志：keepalived服务关闭，vrrp ip被移除 Nov 1 20:28:17 localhost Keepalived[36349]: Stopping Keepalived v1.2.13 (03/19,2015) Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) sending 0 priority Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 1 20:28:17 localhost Keepalived_healthcheckers[36351]: Netlink reflector reports IP 10.1.7.19 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 以下为备用节点日志：转换为MASTER角色，配置10.1.7.19的vrrp ip Nov 1 20:28:18 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:28:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:28:24 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever -
恢复主节点，启动主节点keepalived服务后
[root@ _8_ ~]# service keepalived start Starting keepalived: [ OK ] 以下为主节点日志，转换为MASTER角色，抢占10.1.7.19的ip Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:34:20 localhost Keepalived_healthcheckers[36430]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:34:22 localhost ntpd[2238]: Listen normally on 10 eth0 10.1.7.19 UDP 123 Nov 1 20:34:25 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 以下为备用节点日志，转换为BACKUP角色，ip 10.1.7.19被移除 Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Received higher prio advert Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering BACKUP STATE Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 1 20:34:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever
- keepalived 双主模型(10.1.7.19，10.1.7.20)
-
在单主模型的基础上，主机1再增加一个vrrp_instance段，改动的地方为
vrrp_instance VI_2 { #vrrp示例编号要修改，不能与其他实例相同 state BACKUP #初始状态，上一个实例为MASTER，此时这里为BACKUP interface eth0 virtual_router_id 17 priority 98 #优先级要比另一台主机MASTER的优先级低 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b9 #认证字符串要修改 } virtual_ipaddress { 10.1.7.20/16 dev eth0 #配置的另一个ip要修改 } } -
与之对应的另一台主机2上增加的配置为
vrrp_instance VI_2 { state MASTER interface eth0 virtual_router_id 17 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b9 } virtual_ipaddress { 10.1.7.20/16 dev eth0 } }
- 测试
-
两台主机先停止keepalived服务，然后先启动主机1
以下为主机1日志：VI_1启动为MASTER角色，配置10.1.7.19的ip，VI_2启动为MASTER角色，配置10.1.7.20的ip Nov 1 20:57:42 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:57:43 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:57:45 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Transition to MASTER STATE Nov 1 20:57:45 localhost ntpd[2238]: Listen normally on 11 eth0 10.1.7.19 UDP 123 Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering MASTER STATE Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) setting protocol VIPs. Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 Nov 1 20:57:46 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 added Nov 1 20:57:47 localhost ntpd[2238]: Listen normally on 12 eth0 10.1.7.20 UDP 123 Nov 1 20:57:48 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:57:51 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet 10.1.7.20/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever -
启动主机2
以下为主机1日志：VI_2转换为BACKUP角色，10.1.7.20的ip被移除： Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Received higher prio advert Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering BACKUP STATE Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) removing protocol VIPs. Nov 1 21:03:36 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 以下为主机2的日志，VI_2转换为MASTER角色，配置了10.1.7.20的ip Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Transition to MASTER STATE Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Received lower prio advert, forcing new election Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Entering MASTER STATE Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) setting protocol VIPs. Nov 1 21:03:37 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.20 added Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 Nov 1 21:03:42 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.20/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever -
停止主机1的keepalived服务。
以下为主机2日志：VI_1转换为MASTER角色，配置了10.1.7.19的ip Nov 1 21:07:47 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 21:07:48 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.19 added Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 21:07:53 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.20/16 scope global secondary eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever
3. Keepalived集群+ipvs（DR）集群
拓扑环境
10.1.6.11和10.1.6.12为两台real server，提供web服务。
左边两台服务器，主节点ip10.1.6.11，备节点ip10.1.6.12
主节点和备节点做成keepalived高可用集群。IP为10.1.8.88
-
两台real server 安装httpd，编辑测试主页，启动httpd服务，在主节点或备用节点上请求测试主页
[root@ _2_ ~]# yum -y install httpd [root@ _2_ ~]# cat /var/www/html/index.html <h1>Server 1</h1> [root@ _2_ ~]# yum -y install httpd [root@ _2_ ~]# cat /var/www/html/index.html <h1>Server 2</h1> [root@ _3_ ~]# curl http://10.1.7.11 <h1>Server 1</h1> [root@ _4_ ~]# curl http://10.1.7.12 <h1>Server 2</h1> -
Real Server上编写DR模型初始配置脚本，两台real server运行此脚本
#!/bin/bash vip='10.1.8.88' vport='80' netmask='255.255.255.255' iface='lo:0' case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $iface $vip netmask $netmask broadcast $vip up route add -host $vip dev $iface ;; stop) ifconfig $iface down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage $(basename $0) start|stop" exit 1 ;; esac 检查配置 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 10.1.8.88 netmask 255.255.255.255 loop txqueuelen 0 (Local Loopback) -
主节点和备用节点上配置keepalived
以下为主节点配置，备用节点需修改state为BACKUP ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from Keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.0.200.158 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 16 priority 98 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b8 } virtual_ipaddress { 10.1.8.88/16 dev eth0 } } -
测试主备节点分别故障时，虚拟ip能来回切换
-
主节点和备节点安装ipvsadm，测试调度后端real server，确保调度正常
主节点： [root@ _8_ ~]# yum -y install ipvsadm [root@ _8_ ~]# ipvsadm -A -t 10.1.8.88:80 -s rr [root@ _9_ ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.11 -g -w 1 [root@ _10_ ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.12 -g -w 1 [root@ _11_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 [root@ _13_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> 调度正常，清空规则 [root@ _25_ ~]# ipvsadm -C 备用节点同样方法测试一次 -
定义keepalived配置中Virtual Server
主节点和备节点的keepalived配置文件中加入Virtual server配置段 virtual_server 10.1.8.88 80 { #virtual_server ip地址 delay_loop 3 #服务轮询时间间隔 lb_algo rr #定义调度算法 lb_kind DR #定义lvs的类型 protocol TCP #服务协议，仅支持tcp real_server 10.1.7.11 80 { #real_server ip地址 weight 1 #权重 HTTP_GET { #请求方法 url { path / #定义监控的url status_code 200 #判断上述检测机制为健康状态的响应码为200 } connect_timeout 1 #连接超时时间 nb_get_retry 3 #重试的次数 delay_before_retry 1 #重试之前延迟时长 } } real_server 10.1.7.12 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } -
启动主节点和备节点keepalived服务，查看ip以及ipvs规则
主节点 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever [root@ _33_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 备节点 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever [root@ _28_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 -
使用客户端对其进行访问检测
测试正常 [root@ _16_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> -
使real server中有一个故障，检测访问
停掉real server的httpd服务 [root@ _5_ ~]# systemctl stop httpd 主节点上查看ipvs规则，real server 2已下线 [root@ _38_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 10 客户端请求访问 [root@ _17_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> 访问正常 -
使故障的real server恢复，检测访问
[root@ _6_ ~]# systemctl start httpd 查看主节点上ipvs规则，real server已加入 [root@ _39_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 客户端访问测试 [root@ _18_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> 调度正常
4. keepalived主备节点上配置sorry server
-
主备节点分别安装httpd，编辑网页文件，最好都停掉keepalived服务
[root@ _41_ ~]# yum -y install httpd 主节点编辑网页文件 [root@ _38_ ~]# cat /var/www/html/index.html <h1>LB Cluster Fault,this is Sorry Server 1</h1> 主节点编辑网页文件 [root@ _38_ ~]# cat /var/www/html/index.html <h1>LB Cluster Fault,this is Sorry Server 2</h1> -
编辑keepalived配置文件，在Virtual server中添加sorry server配置，主节点和备节点都要配置
virtual_server 10.1.8.88 80 { delay_loop 3 lb_algo rr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 10.1.7.11 80 { weight 1 ... -
主备节点都启动httpd服务，启动keepalived服务，两台real server都停止httpd服务
[root@ _48_ ~]# service httpd start [root@ _44_ ~]# service keepalived start Starting keepalived: [ OK ] [root@ _12_ ~]# systemctl stop httpd -
客户端请求测试
看到sorry server的响应 [root@ _22_ ~]# curl http://10.1.8.88 <h1>LB Cluster Fault,this is Sorry Server 1</h1> -
启动一台real server的httpd服务，客户端测试
响应正常 [root@ _23_ ~]# curl http://10.1.8.88 <h1>Server 2</h1>
5.keeplioved调用外部脚本，由结果实时调整优先级
-
脚本的定义与调用介绍
-
（1）脚本先定义 vrrp_script<SCRIPT_NAME> { script "" interval INT weight -INT } （2）再调用 track_script { SCRIPT_NAME_1 SCRIPT_NAME_2 ... } -
主备节点keepalived配置加入脚本段，脚本检测到/etc/keepalived/down则返回失败
主节点 ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from Keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.0.200.158 } vrrp_script chk_down { #定义脚本名 script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #判断down文件，存在则返回失败 interval 1 #每隔多长时间脚本执行一次 weight -5 #脚本失败动作，权重-5，确保-5后低于备用优先级 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 16 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b8 } virtual_ipaddress { 10.1.8.88/16 dev eth0 } track_script { #监控的脚本 chk_down #调用的脚本名称 } } 备节点需要修改state为BACKUP，priority为98 -
主备节点启动keepalived服务，查看ip
[root@ _72_ /etc/keepalived]# service keepalived stop Starting keepalived: [ OK ] 主节点 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 备节点 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 -
创建/etc/keepalived/down文件，查看ip转移
[root@ _161_ /etc/keepalived]# touch down 主节点 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link 备节点 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link 主节点检测脚本，返回失败，权重-5，转换为BACKUP角色，ip 10.1.8.88被移除 Nov 3 08:24:02 localhost Keepalived_vrrp[4853]: VRRP_Script(chk_down) failed Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Received higher prio advert Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Entering BACKUP STATE Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 3 08:24:03 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 removed -
删除主节点/etc/keepalived/down文件，查看ip转移
[root@ _163_ /etc/keepalived]# rm -rf down 主节点，ip已夺回 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link 日志 Nov 3 08:32:01 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 added Nov 3 08:32:01 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.8.88 Nov 3 08:32:03 localhost ntpd[4558]: Listen normally on 11 eth0 10.1.8.88 UDP 123
6. keepalived结合nginx调度（并使用辅助脚本检测nginx服务）
-
清空上例在real server上所做的lo：0接口规则以及ARP限制规则，并停止keepalived服务
[root@ _16_ ~]# bash set_dr stop #!/bin/bash vip='10.1.8.88' vport='80' netmask='255.255.255.255' iface='lo:0' case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $iface $vip netmask $netmask broadcast $vip up route add -host $vip dev $iface ;; stop) ifconfig $iface down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage $(basename $0) start|stop" exit 1 ;; esac -
主备节点停止为sorry server启动的httpd服务
[root@ _50_ ~]# service httpd stop Stopping httpd: [ OK ] -
主备节点安装nginx
[root@ _173_ /etc/keepalived]# yum -y install nginx -
编辑nginx配置文件，实现反代
在/etc/nginx/nginx.conf的http上下文中添加 upstream websrvs { server 10.1.7.11; server 10.1.7.12; } 在/etc/nginx/conf.d/default.conf的location上下文中添加 proxy_pass http://websrvs; 如 location / { root /usr/share/nginx/html; proxy_pass http://websrvs; index index.html index.htm; } -
主备节点启动nginx服务，keepalived服务，访问测试
[root@ _18_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _19_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _20_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _21_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> 访问正常 -
主备节点添加vrrp_script脚本，在nginx没有启动时触发
vrrp_script chk_nginx { script "killall -0 nginx && exit 0 || exit 1" interval 1 weigth } 追踪脚本中也需要加入chk_nginx track_script { chk_down chk_nginx } -
主节点重启keepalived服务，而后备节点重启keepalived服务
此时虚拟ip 10.1.8.88在主节点上 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 客户端访问正常 [root@ _22_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _23_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _24_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> -
停止主节点nginx服务
[root@ _12_ ~]# service nginx stop Stopping nginx: [ OK ] 主节点ip已移除 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 日志信息，检测到vrrp_script失败，转换为BACKUP模式，ip已移除 Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Script(chk_nginx) failed Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Entering FAULT STATE Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Now in FAULT state Nov 3 18:00:25 localhost Keepalived_healthcheckers[75163]: Netlink reflector reports IP 10.1.8.88 removed 备节点ip与日志，10.1.8.88地址已获取，转换为MASTER角色 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link Nov 3 18:00:26 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 3 18:00:27 localhost Keepalived_healthcheckers[75083]: Netlink reflector reports IP 10.1.8.88 added -
客户端访问测试
调度正常 [root@ _25_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _26_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _27_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _28_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1>
补充:keepalived发生角色转移时运行指定脚本
-
备节点上编写脚本/etc/keepalived/motify.sh，当角色切换时，给root用户发送邮件
#!/bin/bash # contact='root@localhost' notify() { mailsubject="$(hostname) to be $1, vip floating." mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac -
在vrrp_instance端中调用脚本，并重启keepalived服务
notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault"
3.主节点停止keepalived服务，查看root用户邮件
备节点：收到转换为master角色的邮件
>N 1 root Thu Nov 3 18:41 18/731 "localhost.localdomain to be master, vip floating."
& 1
Message 1:
From root@localhost.localdomain Thu Nov 3 18:41:46 2016
Return-Path: <root@localhost.localdomain>
X-Original-To: root@localhost
Delivered-To: root@localhost.localdomain
Date: Thu, 03 Nov 2016 18:41:46 +0800
To: root@localhost.localdomain
Subject: localhost.localdomain to be master, vip floating.
User-Agent: Heirloom mailx 12.4 7/29/08
Content-Type: text/plain; charset=us-ascii
From: root@localhost.localdomain (root)
Status: R
2016-11-03 18:41:46: vrrp transition, localhost.localdomain changed to be master
ip 10.1.8.88已添加
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
