LVS+Keepalived高可用集群(CentOS7)
Keepalived高可用集群
什么是高可用集群？
keepalived是集群管理中保证集群高可用的一个服务软件，其功能类似于heartbeat，用来防止单点故障。
keepalived故障切换转移原理
keepalived是以VRRP协议为实现基础的，VRRP虚拟路由冗余协议，可以认为是实现路由器高可用的协议，即将N台提供相同功能的路由器组成一个路由器组，这个组里面有一个master和多个backup，master上面有一个对外提供服务的vip（该路由器所在局域网内其他机器的默认路由为该vip），master会发组播，当backup收不到vrrp包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master。这样的话就可以保证路由器的高可用了。
keepalived主要的三个模块 core、check和vrrp
1）core模块为keepalived的核心，负责主进程的启动、维护以及全局配置文件的加载和解析。
2）check负责健康检查，包括常见的各种检查方式。
3）vrrp模块是来实现VRRP协议的。
VRRP协议的简单介绍
VRRP（Virtual Router Redundancy Protocol），即虚拟路由冗余协议。VRRP的出现就是为解决静态路由单点故障，通过一种竞选协议机制来将路由任务交给某台VRRP路由器
VRRP所有报文都是通过IP多播（multicast）包（多播地址:224.0.018）形成发送的，虚拟路由器由VRID（范围0-255）和一组IP地址组成，对外表现为一个周知的MAC地址：00-00-5E-00-01{VRID}，所以，在一个虚拟路由中，不管谁是MASTER，对外都是相同的MAC和IP（也就是VIP），客户端主机并不需要因为MASTER的改变而修改自己的路由配置，对他们来说，主从切换是透明
在虚拟路由中，只有MASTER的VRRP路由会一直发送VRRP广告包（VRRP Advertisement message）,BACKUP不会抢占MASTER，除非它的优先级（priority）比MASTER的更高 ，当MASTER不可用时，BACKUP收不到广告包，多台BACKUP优先级最高的就会抢占为MASTER，这种抢占很快速（一般小于1s），以保证服务的连续性
keepalived的配置文件
1、keepalived的配置文件可以实现LVS功能
2、keepalived可以对LVS下面的集群节点做健康检查
keepalived只有一个配置文件keepalived.conf，里面主要包括以下几个配置区域，分别是global_defs、static_ipaddress、static_routes、vrrp_script、vrrp_instance和virtual_server。
keepalived 服务两大用途：ha failover & healthcheck
ha failover：主要实现LB master 和 backup 主机之间故障转移和自动转换，当主负载均衡器（MASTER）失效或出现故障时，BACKUP将自动接管MASTER的工作（VIP资源及相应服务），一旦MASTER负载均衡器修复，master又会接管回他的工作
rs healthcheck：负载均衡定期检查RS的可用性决定是否给其分发请求：当虚拟服务器中的某一个甚至几个真实服务器同时发生故障无法提供服务时，负载均衡器会自动将失效的RS服务器清除出去，从而保证用户访问不会受到影响，当故障的RS修复后，系统又自动把加入转发队列，分发请求提供正常服务
keepalived小结
VRRP协议介绍
1、VRRP（Virtual Router Redundancy Protocol），即虚拟路由冗余协议。VRRP的出现就是解决了静态路由单点故障
2、VRRP是通过一种竞选协议机制来将路由任务交给某台VRRP路由器
3、VRRP通信是用IP多播的方式实现的
4、MASTER发包、BACKUP收包，当BACKUP收不到包时，优先级最高的BACKUP竞选为MASTER
5、VRRP使用了加密协议
LVS+Keepalived环境部署准备
环境准备：两台keepalived服务器
keepalived服务器 1： 内网IP：10.10.10.11 外网IP：20.20.20.11
keepalived服务器 2： 外网IP：10.10.10.12 外网IP：20.20.20.12
真实服务器1： 内网IP：10.10.10.13
真实服务器2： 内网IP：10.10.10.14
LVS+Keepalived调度器网络环境部署
开启并启用网卡子接口
1、关闭网卡守护进程
systemctl stop NetworkManager
systemctl disable NetworkManager
2、开启子接口
[root@LVS2 ~]# cd /etc/sysconfig/network-scripts/
[root@LVS2 network-scripts]# cp -a /etc/sysconfig/network-scripts/{ifcfg-lo,ifcfg-lo:0}
[root@LVS2 network-scripts]# vim ifcfg-lo:0
DEVICE=lo:0
IPADDR=10.10.10.100
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback
3、启用子接口
[root@WEB1 network-scripts]# ifup ifcfg-lo:0
4、修改内核参数
[root@LVS2 ~]# vim /etc/sysctl.conf
# LVS-DR 网卡重定向
net.ipv4.conf.all.send_redirects = 0 #禁止转发重定向报文
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.lo.send_redirects = 0
[root@LVS2 ~]# sysctl -p
sysctl: setting key "net.ipv4.conf.all.send_redirects": Invalid argument
net.ipv4.conf.all.send_redirects = 0 #禁止转发重定向报文
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.lo.send_redirects = 0
真实服务器网络环境配置
开机并启用网卡子接口
1、关闭网卡守护进程
systemctl stop NetworkManager
systemctl disable NetworkManager
2、配置回环子接口
[root@WEB1 ~]# cd /etc/sysconfig/network-scripts/
[root@WEB1 network-scripts]# cp -a ifcfg-lo ifcfg-lo:0
[root@WEB1 network-scripts]# vim ifcfg-lo:0
DEVICE=lo:0
IPADDR=10.10.10.100
NETMASK=255.255.255.255
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
[root@WEB1 network-scripts]# ifup ifcfg-lo:0
[root@WEB1 network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.13 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::a2d:9ece:c698:b051 prefixlen 64 scopeid 0x20<link>
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 10.10.10.100 netmask 255.255.255.255
loop txqueuelen 1 (Local Loopback)
3、修改内核参数
[root@WEB1 ~]# vim /etc/sysctl.conf
#LVS-DR ARP通信行为控制
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@WEB1 ~]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
4、添加路由
[root@WEB1 ~]# route add -host 10.10.10.100 dev lo:0
[root@WEB2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 ens33
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
10.10.10.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
5、将路由添加到开机自启动
[root@WEB1 ~]# echo "lvs-dr路由记录" >> /etc/rc.local
[root@WEB1 ~]# echo "route add -host 10.10.10.100 dev lo:0" >> /etc/rc.local
6、启用httpd服务
[root@WEB1 ~]# yum -y install httpd
[root@WEB1 ~]# echo "This is RS1" >> /var/www/html/index.html
[root@WEB1 ~]# systemctl start httpd
[root@WEB1 ~]# systemctl disable httpd
[root@WEB1 ~]# curl 127.0.0.1
This is RS1
Keepalived安装
一、下载keepalived
二、安装依赖包
[root@LVS2 ~]# yum -y install kernel-devel popt-devel gcc* openssl-devel libnl3-devel pcre-devel net-snmp-devel libnfnetlink-devel
三、安装keepalived
[root@LVS1 ~]# tar -xf keepalived-2.0.19.tar.gz
[root@LVS1 ~]# tar -xf rkhunter-1.4.6.tar.gz
[root@LVS1 ~]# cd keepalived-2.0.19/
[root@LVS1 kernels]# ./configure --prefix=/ --with-kernel=/usr/src/kernels/3.10.0-1062.el7.x86_64
[root@LVS1 keepalived-2.0.19]# make && make install
[root@LVS1 keepalived-2.0.19]# cp -a /root/keepalived-2.0.19/keepalived/etc/init.d/keepalived /etc/init.d/
[root@LVS1 keepalived-2.0.19]# cp -a /root/keepalived-2.0.19/keepalived/keepalived.service /etc/systemd/system/
[root@LVS1 ~]# chmod -R 755 /etc/init.d/keepalived #给keepalived启动程序赋权
[root@LVS1 ~]# systemctl enable keepalived.service #将keepalived设置为开机自启
[root@LVS1 ~]# systemctl start keepalived.service
[root@LVS1 keepalived]# vim /var/run/keepalived.pid
[root@LVS1 keepalived]# vim /lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-//etc/sysconfig/keepalived
ExecStart=//sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
[root@LVS1 ~]# systemctl daemon-reload
四、配置keepalived.conf配置文件
主keepalived调度器配置
[root@LVS1 network-scripts]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id R1 #两边可以不一样
}
vrrp_instance VI_1 {
state MASTER #配置MASTER和BACKUP
interface ens32 #心跳网卡
virtual_router_id 51 #虚拟器路由器ID，主备要一致
priority 100 #优先级
advert_int 1 #检查心跳间隔时间，单位秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.10.100/24 dev ens32 #VIP和工作端口号
}
}
virtual_server 10.10.10.100 80 { #LVS配置，VIP
delay_loop 3 #服务论询的时间间隔
lb_algo rr #LVS调度算法
lb_kind DR #LVS集群模式
protocol TCP
real_server 10.10.10.13 80 { #真实服务器IP
weight 1
TCP_CHECK {
connect port 80
connect timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
real_server 10.10.10.14 80 {
weight 1
TCP_CHECK {
connect port 80
connect timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
主keepalived调度器配置
! Configuration File for keepalived
global_defs {
router_id R2 #两边可以不一样
}
vrrp_instance VI_1 {
state BACKUP #配置MASTER和BACKUP
interface ens33 #心跳网卡
virtual_router_id 51 #虚拟器路由器ID，主备要一致
priority 80 #优先级
advert_int 1 #检查心跳间隔时间，单位秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.10.100/24 dev ens33 #VIP和工作端口号
}
}
virtual_server 10.10.10.100 80 { #LVS配置，VIP
delay_loop 3 #服务论询的时间间隔
lb_algo rr #LVS调度算法
lb_kind DR #LVS集群模式
protocol TCP
real_server 10.10.10.13 80 { #真实服务器IP
weight 1
TCP_CHECK {
connect port 80
connect timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
real_server 10.10.10.14 80 {
weight 1
TCP_CHECK {
connect port 80
connect timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
五、配置keepalived服务
[root@LVS1 ~]# systemctl start keepalived.service
[root@LVS1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.10.100:80 rr
-> 10.10.10.13:80 Route 1 0 2
-> 10.10.10.14:80 Route 1 0 2
测试
一、测试服务器是否实现负载均衡
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@Admin ~]# curl 10.10.10.100
This is RS1
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@Admin ~]# curl 10.10.10.100
This is RS1
二、测试停止主keepalived，是否可以正常访问
[root@LVS1 ~]# systemctl stop keepalived.service
[root@Admin ~]# curl 10.10.10.100
This is RS1
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@Admin ~]# curl 10.10.10.100
This is RS1
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@LVS2 ~]# ip a #查看负载均衡IP状态，MASTER上有VIP虚拟IP
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:0f:a9:92 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.12/24 brd 10.10.10.255 scope global ens33
valid_lft forever preferred_lft forever
inet 10.10.10.100/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::f3fd:bcb0:3ff8:f5ae/64 scope link
valid_lft forever preferred_lft forever
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:0f:a9:9c brd ff:ff:ff:ff:ff:ff
[root@LVS1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.10.10.100/32 brd 10.10.10.100 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fe:30:9b brd ff:ff:ff:ff:ff:ff
inet 10.10.10.11/24 brd 10.10.10.255 scope global ens32
valid_lft forever preferred_lft forever
inet6 fe80::a939:89f8:1beb:5ca7/64 scope link
valid_lft forever preferred_lft forever
3: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fe:30:a5 brd ff:ff:ff:ff:ff:ff
三、测试主keepalived 恢复正常后，能否抢回MASTER的位置
[root@LVS1 ~]# systemctl start keepalived.service
[root@LVS1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.10.100:80 rr
-> 10.10.10.14:80 Route 1 0 0
[root@LVS1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.10.10.100/32 brd 10.10.10.100 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fe:30:9b brd ff:ff:ff:ff:ff:ff
inet 10.10.10.11/24 brd 10.10.10.255 scope global ens32
valid_lft forever preferred_lft forever
inet 10.10.10.100/24 scope global secondary ens32
valid_lft forever preferred_lft forever
inet6 fe80::a939:89f8:1beb:5ca7/64 scope link
valid_lft forever preferred_lft forever
3: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fe:30:a5 brd ff:ff:ff:ff:ff:ff
当主Keepalived恢复正常后，立即变为MASTER
三、当一台真实服务器停止服务后，测试是否能正常访问
[root@WEB1 ~]# systemctl stop httpd
[root@LVS1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.10.100:80 rr
-> 10.10.10.14:80 Route 1 0 0
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@Admin ~]# curl 10.10.10.100
This is RS2
[root@Admin ~]# curl 10.10.10.100
This is RS2
