LVS负载均衡(3)-- DR模型搭建实例
1. LVS DR模型搭建
1.1 DR模型网络规划
规划要点:
- 在生产环境中,客户端与企业互联网出口设备不会时同一网段地址,此处我们规划为同一网段地址,但是没有在客户端上配置网关,因此我们需要在企业出口设备上把LVS的VIP地址192.168.50.100做地址映射或端口映射,映射到企业出口防火墙的外网地址192.168.20.31。
- LVS设备在LAN网络中设置了两个IP,其中192.168.50.31为实际地址DIP,192.168.50.100为虚IP,为后面的高可用规划地址。
- 后端nginx服务器RS设备与LVS在同一网段,网关无需指向LVS设备,指向出口路由器即可。
- 由于CIP请求的是VIP,而响应是通过RIP响应给CIP,所以数据报文一定会被丢弃。那么就需要在所有的RS的接口上配置VIP的地址。由RS上的VIP响应给CIP即可。
- 所有的客户端都是与后端的RS主机进行TCP三次握手,而不是LVS设备。
- DR模型下LVS设备无需打开ip_forward功能。
1.2 RS设备的VIP冲突解决方式
RS设备上配置VIP会引起地址冲突,同时路由器有可能会直接把请求发给RS设备,而不是LVS,解决方法有以下几种:
-
在前端路由器做arp静态绑定,绑定VIP和Director的MAC地址。
-
在RS上使用arptables工具:
arptables -A IN -d $VIP -j DROP
arptables -A OUT -s $VIP -j mangle --mangle-ip-s $RIP
-
在RS上修改内核参数以限制arp通告及应答级别(lo和all都要修改),保存配置需要修改配置文件/etc/sysctl.conf。
-
1.不回应广播
arp_ignore=1,找到设定目录find /proc -name "*arp_ignore",修改lo和all目录下的项
0:默认值,表示可使用本地任意接口上配置的任意地址进行响应
1:仅在请求的目标IP配置在本地主机接收到请求报文的接口上时,才给予响应
-
2.启动时不发免费ARP
arp_announce=2,找到设定目录find /proc -name "*arp_announce",修改lo和all目录下的项
0:默认值,把本机所有接口的所有信息向每个接口的网络进行通告
1:尽量避免将接口信息向非直接连接网络进行通告
2:必须避免将接口信息向非本网络进行通告
-
然后把VIP地址设置在RS设备的lo0:0接口上。
-
1.3 DR模型访问流程
DR模型的访问流程如下:
- 1、当用户请求到达 DS节点 ,此时请求的数据报文会先到内核空间的PREROUTING 链。 此时报文的 源IP为CIP , 目标IP为VIP 。
- 2、 PREROUTING 检查发现数据包的 目标IP 是本机,将数据包送至 INPUT 链。
- 3、 IPVS 比对数据包请求的服务是否为集群服务,是则将请求报文中的 源MAC 修改为 DMAC ,将 目标MAC 修改 RMAC ,然后将数据包通过 POSTROUTING 链发出。此时的 源IP 和 目的IP 均未修改,仅将 源MAC 修改为 DMAC , 目标MAC 修改为 RMAC
- 4、由于 DS 和 RS 在同一个网络中,所以是通过二层来传输。 POSTROUTING 链检查 目标MAC 为 RIP的MAC 地址,那么此时数据包将通过 DIP 发送 RS 节点
- 5、 RS 拆解数据报文发现请求的 IP 地址是本机,则会接收该数据报文,而后构建响应报文向外发出,此时的 源IP 是 VIP , 目标IP 是 CIP
- 6、响应报文最终送达至客户端
1.4 DR模型配置
1.4.1 ROUTER设备配置
-
ROUTER设备的IP地址和路由信息如下:
[root@router ~]# ip add 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:4f:a9:ca brd ff:ff:ff:ff:ff:ff inet 192.168.20.50/24 brd 192.168.20.255 scope global noprefixroute eth1 valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:4f:a9:d4 brd ff:ff:ff:ff:ff:ff inet 192.168.50.50/24 brd 192.168.50.255 scope global noprefixroute eth2 valid_lft forever preferred_lft forever #此场景中无需配置路由 [root@router ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.20.0 0.0.0.0 255.255.255.0 U 101 0 0 eth1 192.168.50.0 0.0.0.0 255.255.255.0 U 104 0 0 eth2
-
打开router设备的ip_forward功能:
[root@router ~]# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf [root@router ~]# sysctl -p net.ipv4.ip_forward = 1
-
把LVS的虚IP地址的80和443端口映射到路由器外网地址的80和443端口,也可以使用地址映射:
#端口映射: [root@router ~]# iptables -t nat -A PREROUTING -d 192.168.20.50 -p tcp --dport 80 -j DNAT --to 192.168.50.100:80 [root@router ~]# iptables -t nat -A PREROUTING -d 192.168.20.50 -p tcp --dport 443 -j DNAT --to 192.168.50.100:443 #地址映射: [root@router ~]# iptables -t nat -A PREROUTING -d 192.168.20.50 -j DNAT --to 192.168.50.100 #源NAT,让内部主机上网使用 [root@router ~]# iptables -t nat -A POSTROUTING -s 192.168.50.0/24 -j SNAT --to 192.168.20.50 #查看NAT配置: [root@router ~]# iptables -t nat -vnL Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0.0.0.0/0 192.168.20.50 tcp dpt:80 to:192.168.50.100:80 0 0 DNAT tcp -- * * 0.0.0.0/0 192.168.20.50 tcp dpt:443 to:192.168.50.100:443 Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * * 192.168.50.0/24 0.0.0.0/0 to:192.168.20.50
1.4.2 后端nginx服务器配置
-
nginx02主机的网络配置如下:
#1.在lo接口配置VIP地址: [root@nginx02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-lo:0 DEVICE=lo:0 BOOTPROTO=none IPADDR=192.168.50.100 NETMASK=255.255.255.255 <==注意:此处的掩码不能与RIP的掩码配置的一样,否则其他主机无法学习到RIP的ARP信息,会影响RIP的直连路由,而且设置的掩码不能过大,让VIP和CIP计算成同一网段,建议设置为32位掩码。 ONBOOT=yes NAME=loopback #2.重启网卡生效: [root@nginx02 ~]# ifdown lo:0 && ifup lo:0 [root@nginx02 ~]# ifconfig lo:0 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.50.100 netmask 255.255.255.255 loop txqueuelen 1000 (Local Loopback) #3.eth2接口地址如下: [root@nginx02 ~]# ip add 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:d9:f9:7d brd ff:ff:ff:ff:ff:ff inet 192.168.50.22/24 brd 192.168.50.255 scope global noprefixroute eth2 valid_lft forever preferred_lft forever #4.路由配置:网关指向路由器192.168.50.50 [root@nginx02 ~]# ip route add default via 192.168.50.50 dev eth2 <==默认路由必须指定下一跳地址和出接口,否则有可能会从lo:0接口出去,导致不通。 [root@nginx02 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.50.50 0.0.0.0 UG 0 0 0 eth2 192.168.50.0 0.0.0.0 255.255.255.0 U 103 0 0 eth2
-
配置 arp ,不对外宣告本机 VIP 地址,也不响应其他节点发起 ARP 请求 本机的VIP
[root@nginx02 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@nginx02 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore [root@nginx02 ~]# echo 1 > /proc/sys/net/ipv4/conf/default/arp_ignore [root@nginx02 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce [root@nginx02 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce [root@nginx02 ~]# echo 2 > /proc/sys/net/ipv4/conf/default/arp_announce
-
nginx03主机的网络配置如下:
#1.在lo接口配置VIP地址: [root@nginx03 ~]# cat /etc/sysconfig/network-scripts/ifcfg-lo:0 DEVICE=lo:0 BOOTPROTO=none IPADDR=192.168.50.100 NETMASK=255.255.255.255 <==注意:此处的掩码不能与RIP的掩码配置的一样,否则其他主机无法学习到RIP的ARP信息,会影响RIP的直连路由,而且设置的掩码不能过大,让VIP和CIP计算成同一网段,建议设置为32位掩码。 ONBOOT=yes NAME=loopback #2.重启网卡生效: [root@nginx03 ~]# ifdown lo:0 && ifup lo:0 [root@nginx03 ~]# ifconfig lo:0 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.50.100 netmask 255.255.255.255 loop txqueuelen 1000 (Local Loopback) #3.eth2接口地址如下: [root@nginx03 ~]# ip add show eth2 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:0a:bf:63 brd ff:ff:ff:ff:ff:ff inet 192.168.50.23/24 brd 192.168.50.255 scope global noprefixroute eth2 valid_lft forever preferred_lft forever #4.路由配置:网关指向路由器192.168.50.50 [root@nginx03 ~]# ip route add default via 192.168.50.50 dev eth2 <==默认路由必须指定下一跳地址和出接口,否则有可能会从lo:0接口出去,导致不通。 [root@nginx03 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.50.50 0.0.0.0 UG 0 0 0 eth2 192.168.50.0 0.0.0.0 255.255.255.0 U 103 0 0 eth2
-
配置 arp ,不对外宣告本机 VIP 地址,也不响应其他节点发起 ARP 请求 本机的VIP
[root@nginx03 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@nginx03 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore [root@nginx03 ~]# echo 1 > /proc/sys/net/ipv4/conf/default/arp_ignore [root@nginx03 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce [root@nginx03 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce [root@nginx03 ~]# echo 2 > /proc/sys/net/ipv4/conf/default/arp_announce
-
nginx配置文件两台WEB服务器保持一致:
[root@nginx03 ~]# cat /etc/nginx/conf.d/xuzhichao.conf server { listen 80 default_server; listen 443 ssl; server_name www.xuzhichao.com; access_log /var/log/nginx/access_xuzhichao.log access_json; charset utf-8,gbk; #SSL配置 ssl_certificate_key /apps/nginx/certs/www.xuzhichao.com.key; ssl_certificate /apps/nginx/certs/www.xuzhichao.com.crt; ssl_session_cache shared:ssl_cache:20m; ssl_session_timeout 10m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; keepalive_timeout 65; #防盗链 valid_referers none blocked server_names *.b.com b.* ~\.baidu\. ~\.google\.; if ( $invalid_referer ) { return 403; } client_max_body_size 10m; #浏览器图标 location = /favicon.ico { root /data/nginx/xuzhichao; } location / { root /data/nginx/xuzhichao; index index.html index.php; #http自动跳转https if ($scheme = http) { rewrite ^/(.*)$ https://www.xuzhichao.com/$1; } } } #重启nginx服务: [root@nginx03 ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@nginx03 ~]# systemctl reload nginx.service
-
nginx02主机的主页文件如下:
[root@nginx02 certs]# cat /data/nginx/xuzhichao/index.html node1.xuzhichao.com page
-
nginx03主机的主页文件如下:
[root@nginx03 ~]# cat /data/nginx/xuzhichao/index.html node2.xuzhichao.com page
-
测试访问:
[root@lvs-01 ~]# curl -Hhost:www.xuzhichao.com -k https://192.168.50.23 node2.xuzhichao.com page [root@lvs-01 ~]# curl -Hhost:www.xuzhichao.com -k https://192.168.50.22 node1.xuzhichao.com page
1.4.3 LVS设备配置
-
LVS设备的网络配置如下:
#1.配置虚地址192.168.50.100 #临时配置: [root@lvs-01 ~]# ifconfig eth2:1 192.168.50.100/24 up #永久配置,使用配置文件: [root@lvs-01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2:1 TYPE=Ethernet BOOTPROTO=none IPADDR=192.168.50.100 PREFIX=24 DEVICE=eth2:1 NAME=eth2:1 DEFROUTE=yes ONBOOT=yes #重启网卡: [root@lvs-01 ~]# ifdown eth2 && ifup eth2 [root@lvs-01 ~]# ifdown eth2:1 && ifup eth2:1 #2.LVS地址配置: [root@lvs-01 ~]# ip add 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:21:84:9d brd ff:ff:ff:ff:ff:ff inet 192.168.50.31/24 brd 192.168.50.255 scope global noprefixroute eth2 valid_lft forever preferred_lft forever inet 192.168.50.100/24 brd 192.168.50.255 scope global secondary eth2:1 valid_lft forever preferred_lft forever #3.LVS配置默认路由指向出口路由器192.168.50.50 [root@lvs-01 ~]# ip route add default via 192.168.50.50 #本实验场景中存在ETH1口地址为192.168.20.0/24网段,因此需要配置指向客户端的主机路由,生产中不需要 [root@lvs-01 ~]# ip route add 192.168.20.17/32 via 192.168.50.50 [root@lvs-01 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.50.50 0.0.0.0 UG 0 0 0 eth2 192.168.20.0 0.0.0.0 255.255.255.0 U 101 0 0 eth1 192.168.20.17 192.168.50.50 255.255.255.255 UGH 0 0 0 eth2 192.168.50.0 0.0.0.0 255.255.255.0 U 102 0 0 eth2
-
配置IPVS的规则:
#创建80和443两个集群,并添加后端主机: [root@lvs-01 ~]# ipvsadm -A -t 192.168.50.100:80 -s rr [root@lvs-01 ~]# ipvsadm -A -t 192.168.50.100:443 -s rr [root@lvs-01 ~]# ipvsadm -a -t 192.168.50.100:80 -r 192.168.50.22:80 -g [root@lvs-01 ~]# ipvsadm -a -t 192.168.50.100:80 -r 192.168.50.23:80 -g [root@lvs-01 ~]# ipvsadm -a -t 192.168.50.100:443 -r 192.168.50.22:443 -g [root@lvs-01 ~]# ipvsadm -a -t 192.168.50.100:443 -r 192.168.50.23:443 -g [root@lvs-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.50.100:80 rr -> 192.168.50.22:80 Route 1 0 0 -> 192.168.50.23:80 Route 1 0 0 TCP 192.168.50.100:443 rr -> 192.168.50.22:443 Route 1 0 0 -> 192.168.50.23:443 Route 1 0 0
1.4.4 客户端访问测试
-
客户端网络配置如下:
[root@xuzhichao ~]# ip add 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:2f:d0:da brd ff:ff:ff:ff:ff:ff inet 192.168.20.17/24 brd 192.168.20.255 scope global noprefixroute eth1 valid_lft forever preferred_lft forever [root@xuzhichao ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.20.0 0.0.0.0 255.255.255.0 U 101 0 0 eth1
-
测试访问:
#1.测试使用http方式访问,重定向到https [root@xuzhichao ~]# for i in {1..10} ;do curl -k -L -Hhost:www,xuzhichao.com http://192.168.20.50; done node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page #2.测试直接使用https方式访问 [root@xuzhichao ~]# for i in {1..10} ;do curl -k -Hhost:www,xuzhichao.com https://192.168.20.50; done node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page node2.xuzhichao.com page node1.xuzhichao.com page
-
查看LVS的状态信息:
[root@lvs-01 ~]# ipvsadm -Lnc IPVS connection entries pro expire state source virtual destination TCP 01:46 FIN_WAIT 192.168.20.17:43444 192.168.50.100:443 192.168.50.22:443 TCP 01:47 FIN_WAIT 192.168.20.17:43448 192.168.50.100:443 192.168.50.22:443 TCP 01:49 FIN_WAIT 192.168.20.17:43454 192.168.50.100:443 192.168.50.23:443 TCP 01:47 FIN_WAIT 192.168.20.17:43446 192.168.50.100:443 192.168.50.23:443 TCP 01:47 FIN_WAIT 192.168.20.17:43450 192.168.50.100:443 192.168.50.23:443 TCP 01:48 FIN_WAIT 192.168.20.17:43452 192.168.50.100:443 192.168.50.22:443 [root@lvs-01 ~]# ipvsadm -Ln --stats IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes -> RemoteAddress:Port TCP 192.168.50.100:80 4 12 0 720 0 -> 192.168.50.22:80 2 7 0 420 0 -> 192.168.50.23:80 2 5 0 300 0 TCP 192.168.50.100:443 63 441 0 42210 0 -> 192.168.50.22:443 31 226 0 22185 0 -> 192.168.50.23:443 32 215 0 20025 0 [root@lvs-01 ~]# ipvsadm -Ln --rate IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port CPS InPPS OutPPS InBPS OutBPS -> RemoteAddress:Port TCP 192.168.50.100:80 0 0 0 0 0 -> 192.168.50.22:80 0 0 0 0 0 -> 192.168.50.23:80 0 0 0 0 0 TCP 192.168.50.100:443 0 1 0 57 0 -> 192.168.50.22:443 0 0 0 29 0 -> 192.168.50.23:443 0 0 0 29 0
1.5 DR模型的配置脚本
1.5.1 RS设备的配置脚本
[root@lvs-01 ~]# vim lvs_dr.sh
#!/bin/bash
VIP=192.168.50.100
RS1=192.168.50.22
RS2=192.168.50.23
PORT1=80
PORT2=443
DEV=eth2:1
scheduler=rr
NETMASK=32
case $1 in
start)
#永久配置VIP
cat >/etc/sysconfig/network-scripts/ifcfg-${DEV} <<-EOF
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
NAME=${DEV}
DEVICE=${DEV}
ONBOOT=yes
IPADDR=${VIP}
PREFIX=24
EOF
# 启动网卡
ifup ${DEV}
#临时配置VIP
ifconfig eth2:1 ${VIP}/${NETMASK} up
# 配置LVS规则
ipvsadm -C
ipvsadm -A -t ${VIP}:${PORT1} -s ${scheduler}
ipvsadm -a -t ${VIP}:${PORT1} -r ${RS1} -g
ipvsadm -a -t ${VIP}:${PORT1} -r ${RS2} -g
ipvsadm -A -t ${VIP}:${PORT2} -s ${scheduler}
ipvsadm -a -t ${VIP}:${PORT2} -r ${RS1} -g
ipvsadm -a -t ${VIP}:${PORT2} -r ${RS2} -g
;;
stop)
ifdown ${DEV}
rm -f /etc/sysconfig/network-scripts/ifcfg-${DEV}
ipvsadm -C
;;
*)
echo "Usage: sh $0 { start | stop }"
;;
esac
[root@lvs-01 ~]# sh lvs_dr.sh start
[root@lvs-01 ~]# sh lvs_dr.sh stop
1.5.2 DS设备的配置脚本
[root@nginx02 ~]# vim lvs_rs.sh
#!/usr/bin/bash
VIP=192.168.50.100
DEV=lo:0
case $1 in
start)
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "1" >/proc/sys/net/ipv4/conf/default/arp_ignore
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "2" >/proc/sys/net/ipv4/conf/default/arp_announce
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
cat >/etc/sysconfig/network-scripts/ifcfg-${DEV} <<-EOF
DEVICE=lo:0
IPADDR=${VIP}
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback
EOF
ifup ${DEV} # 启动网卡
pkill -0 nginx && systemctl start nginx
;;
stop)
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/default/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/default/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
ifdown ${DEV} # 停止网卡
rm -f /etc/sysconfig/network-scripts/ifcfg-${DEV}
systemctl stop nginx
;;
*)
echo "Usage: sh $0 { start | stop }"
esac
[root@nginx02 ~]# bash lvs_rs.sh start
[root@nginx02 ~]# bash lvs_rs.sh stop