30.第24章 企业级调度器LVS
一.LVS的四种工作模式
LVS相关术语
- DS：Director Server。指的是前端负载均衡器节点
- RS：Real Server。后端真实的工作服务器
- VIP：向外部直接面向用户请求，作为用户请求的目标的IP地址
- DIP：Director Server IP，主要用于和内部主机通讯的IP地址
- RIP：Real Server IP，后端服务器的IP地址
- CIP：Client IP，访问客户端的IP地址
1、LVS NAT模式
LVS NAT原理详解图
LVS NAT原理：用户请求LVS到达director，director将请求的报文的目的IP改为RIP，同时将报文的目标端口也改为realserver的相应端口，最后将报文发送到realserver上，realserver将数据返回给director，director再把数据发送给用户
LVS NAT特性：
- NAT模式修改的是目的ip，直接走的是switch不需要修改mac地址，所以VIP和RIP不需要在同一个网段内
- NAT的包的进出都需要经过LVS，所以LVS可能会成为一个系统的瓶颈问题
2、LVS DR模式
LVS DR原理详解图
LVS DR原理：用户请求LVS到达director，director将请求的报文的目的MAC地址改为后端的realserver的MAC地址，目的IP为VIP(不变)，源IP为client IP地址(不变)，然后director将报文发送到realserver，realserver检测到目的地址为自己本地的VIP，如果在同一网段，将请求直接返回给用户，如果用户跟realserver不在同一个网段，则需要通过网关返回给用户。
LVS DR特性：
- 前端路由将目标地址为VIP报文统统发给Director Server
- RS跟Director Server必须有一个网卡在同一个物理网络中
- 所有的请求报文经由Director Server，但响应报文必须不能进过Director Server
- 所有的real server机器上都有VIP地址
3、LVS TUN原理
LVS TUN原理详解图
LVS TUN原理：用户请求LVS到达director，director通过IP-TUN加密技术将请求报文的包封装到一个新的IP包里面，目的IP为VIP(不变)，然后director将报文发送到realserver，realserver基于IP-TUN解密，然后解析出来包的目的为VIP，检测网卡是否绑定了VIP，绑定了就处理这个包，如果在同一个网段，将请求直接返回给用户，否则通过网关返回给用户；如果没有绑定VIP就直接丢掉这个包
LVS TUN特性：
- TUNNEL必须在所有的realserver上绑定VIP
- realserver直接把包发给client
- 隧道模式运维起来会比较难，所以一般不用
4、LVS FULLNAT模式
LVS FULLNAT报文变化
LVS FULLNAT特性：
- FULLNAT模式也不需要DIP和RIP在同一网段
- FULLNAT和NAT相比的话：会保证RS的回包一定可到达LVS
- FULLNAT需要更新源IP，所以性能正常比NAT模式下降10%****
5、四种模式的比较
- 是否需要VIP和realserver在同一网段
DR模式因为只修改包的MAC地址，需要通过ARP广播找到realserver，所以VIP和realserver必须在同一个网段，也就是说DR模式需要先确认这个IP是否只能挂在这个LVS下面；其他模式因为都会修改目的地址为realserver的IP地址，所以不需要在同一个网段内 - 是否需要在realserver上绑定VIP
realserver在收到包之后会判断目的地址是否是自己的IP
DR模式的目的地址没有修改，还是VIP，所以需要在realserver上绑定VIP
IP TUN模式值是对包重新包装了一层，realserver解析后的包的IP仍然是VIP，所以也需要在realserver上绑定VIP - 四种模式的性能比较
DR模式、IP TUN模式都是在包进入的时候经过LVS，在包返回的时候直接返回给client；所以二者的性能比NAT高
但TUN模式更加复杂，所以性能不如DR
FULLNAT模式不仅更换目的IP还更换了源IP，所以性能比NAT下降10%
性能比较：DR>TUN>NAT>FULLNAT
二.LVS的十种调度算法
1.静态方法
1.1 轮询调度 rr
均等地对待每一台服务器，不管服务器上的实际连接数和系统负载
1.2 加权轮询 wrr
调度器可以自动问询真实服务器的负载情况，并动态调整权值.
1.3 源地址散列调度算法 sh
与目标地址散列调度算法类似，但它是根据源地址散列算法进行静态分配固定的服务器资源。
1.4 目标地址散列调度算法 dh
该算法是根据目标 IP 地址通过散列函数将目标 IP 与服务器建立映射关系，出现服务器不可用或负载过高的情况下，发往该目标 IP 的请求会固定发给该服务器。
2 动态方法
主要根据每RS当前的负载状态及调度算法进行调度Overhead=value 较小的RS将被调度
2.1 最少链接 lc
动态地将网络请求调度到已建立的连接数最少的服务器上
如果集群真实的服务器具有相近的系统性能，采用该算法可以较好的实现负载均衡
Overhead=activeconns*256+inactiveconns
2.2 加权最少链接 wlc
调度器可以自动问询真实服务器的负载情况，并动态调整权值
带权重的谁不干活就给谁分配，机器配置好的权重高
Overhead=(activeconns*256+inactiveconns)/weight
2.3 最少期望延迟 sed
不考虑非活动链接，谁的权重大，优先选择权重大的服务器来接收请求，但权重大的机器会比较忙
Overhead=(activeconns+1)*256/weight
2.4 永不排队 nq
无需队列，如果有realserver的连接数为0就直接分配过去
2.5 基于局部性的最少连接调度算法 lblc
这个算法是请求数据包的目标 IP 地址的一种调度算法，该算法先根据请求的目标 IP 地址寻找最近的该目标 IP 地址所有使用的服务器，如果这台服务器依然可用，并且有能力处理该请求，调度器会尽量选择相同的服务器，否则会继续选择其它可行的服务器
2.6 复杂的基于局部性最少的连接算法 lblcr
记录的不是要给目标 IP 与一台服务器之间的连接记录，它会维护一个目标 IP 到一组服务器之间的映射关系，防止单点服务器负载过高。
三.LVS-NAT模式案例
[root@centos7 ~]# hostnamectl set-hostname rs1
[root@centos7-2 ~]# hostnamectl set-hostname rs2
[root@rs1 ~]# yum -y install httpd;systemctl enable --now httpd;(hostname -I;hostname) > /var/www/html/index.html
[root@rs2 ~]# yum -y install httpd;systemctl enable --now httpd;(hostname -I;hostname) > /var/www/html/index.html
[root@centos8 ~]# curl 10.0.0.7
10.0.0.7
rs1
[root@centos8 ~]# curl 10.0.0.17
10.0.0.17
rs2
[root@rs1 ~]# yum -y install mariadb-server
[root@rs2 ~]# yum -y install mariadb-server
[root@rs1 ~]# cdnet
[root@rs1 network-scripts]# vim ifcfg-eth0
GATEWAY=10.0.0.8
:wq
[root@rs1 network-scripts]# ip route
default via 10.0.0.8 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.7 metric 100
[root@rs2 ~]# cdnet
[root@rs2 network-scripts]# vim ifcfg-eth0
GATEWAY=10.0.0.8
:wq
[root@rs2 network-scripts]# systemctl restart network
[root@rs2 network-scripts]# ip route
default via 10.0.0.8 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.17 metric 100
[root@centos8 ~]# cdnet
[root@centos8 network-scripts]# ls
ifcfg-eth0
[root@centos8 network-scripts]# cp ifcfg-eth0 ifcfg-eth1
[root@centos8 network-scripts]# vim ifcfg-eth1
DEVICE=eth1
NAME=eth1
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.100
PREFIX=24
:wq
[root@centos8 ~]# hostnamectl set-hostname lvs
[root@lvs ~]# reboot
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:3f:19:17 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe3f:1917/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:3f:19:21 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.100/24 brd 192.168.10.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe3f:1921/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu2004:/etc/netplan# vim 01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
eth0:
addresses: [192.168.10.200/24]
:wq
root@ubuntu2004:/etc/netplan# netplan apply
root@ubuntu2004:~# ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.438 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.445 ms
^C
--- 192.168.10.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1017ms
rtt min/avg/max/mdev = 0.438/0.441/0.445/0.003 ms
[root@lvs ~]# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=64 time=0.452 ms
^C
--- 10.0.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.452/0.452/0.452/0.000 ms
[root@lvs ~]# ping 10.0.0.17
PING 10.0.0.17 (10.0.0.17) 56(84) bytes of data.
64 bytes from 10.0.0.17: icmp_seq=1 ttl=64 time=0.700 ms
64 bytes from 10.0.0.17: icmp_seq=2 ttl=64 time=0.287 ms
^C
--- 10.0.0.17 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 27ms
rtt min/avg/max/mdev = 0.287/0.493/0.700/0.207 ms
[root@lvs ~]# ping 192.168.10.200
PING 192.168.10.200 (192.168.10.200) 56(84) bytes of data.
64 bytes from 192.168.10.200: icmp_seq=1 ttl=64 time=0.277 ms
^C
--- 192.168.10.200 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.277/0.277/0.277/0.000 ms
[root@lvs ~]# dnf -y install ipvsadm
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lvs ~]# ipvsadm -A -t 192.168.10.100:80 -s rr
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 rr
[root@lvs ~]# ipvsadm -a -t 192.168.10.100:80 -r 10.0.0.7 -m
[root@lvs ~]# ipvsadm -a -t 192.168.10.100:80 -r 10.0.0.17 -m
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 rr
-> 10.0.0.7:80 Masq 1 0 0
-> 10.0.0.17:80 Masq 1 0 0
root@ubuntu2004:~# curl 192.168.10.100
[root@lvs ~]# tcpdump -i eth1 -nn port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:49.008394 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699730811 ecr 0,nop,wscale 7], length 0
20:49:50.017583 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699731820 ecr 0,nop,wscale 7], length 0
20:49:52.033188 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699733836 ecr 0,nop,wscale 7], length 0
20:49:56.289533 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699738092 ecr 0,nop,wscale 7], length 0
20:50:04.480518 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699746284 ecr 0,nop,wscale 7], length 0
[root@lvs ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward=1
:wq
[root@lvs ~]# sysctl -p
net.ipv4.ip_forward = 1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.17
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
[root@rs2 network-scripts]# vim /etc/httpd/conf/httpd.conf
Listen 8080
:wq
[root@rs2 ~]# systemctl restart httpd
[root@rs2 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 [::]:8080 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*
root@ubuntu2004:~# curl 192.168.10.100
curl: (7) Failed to connect to 192.168.10.100 port 80: Connection refused
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
[root@lvs ~]# systemctl stop ipvsadm
[root@lvs ~]# ipvsadm-save >ipvsadm.log
[root@lvs ~]# cat ipvsadm.log > /etc/sysconfig/ipvsadm
[root@lvs ~]# cat /etc/sysconfig/ipvsadm
-A -t lvs:http -s rr
-a -t lvs:http -r 10.0.0.7:http -m -w 1
-a -t lvs:http -r 10.0.0.17:http -m -w 1
[root@lvs ~]# vim /etc/sysconfig/ipvsadm
-A -t 190.168.10.100:80 -s rr
-a -t 192.168.10.100:80 -r 10.0.0.7:80 -m -w 1
-a -t 192.168.10.100:80 -r 10.0.0.17:8080 -m -w 1:wq
[root@lvs ~]# systemctl start ipvsadm
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 rr
-> 10.0.0.7:80 Masq 1 0 0
-> 10.0.0.17:8080 Masq 1 0 0
[root@lvs ~]# systemctl enable ipvsadm
Created symlink /etc/systemd/system/multi-user.target.wants/ipvsadm.service → /usr/lib/systemd/system/ipvsadm.service.
[root@lvs ~]# reboot
oot@ubuntu2004:~# curl 192.168.10.100
10.0.0.17
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
[root@lvs ~]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 192.168.10.100:80 4 26 8 1636 941
-> 10.0.0.7:80 2 13 4 818 470
-> 10.0.0.17:80 2 13 4 818 471
[root@lvs ~]# systemctl stop ipvsadm
[root@lvs ~]# vim /etc/sysconfig/ipvsadm
-A -t 192.168.10.100:80 -s wrr
-a -t 192.168.10.100:80 -r 10.0.0.7:80 -m -w 3
-a -t 192.168.10.100:80 -r 10.0.0.17:8080 -m -w 1
[root@lvs ~]# systemctl start ipvsadm
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 wrr
-> 10.0.0.7:80 Masq 3 0 1
-> 10.0.0.17:8080 Masq 1 0 1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.17
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.17
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7
rs1
四.LVS-DR模式单网段案例
root@ubuntu2004:~# vim /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
eth0:
addresses: [192.168.10.123/24]
gateway4: 192.168.10.200
:wq
root@ubuntu2004:~# reboot
root@ubuntu2004:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:fb:a9:38 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.123/24 brd 192.168.10.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fefb:a938/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu2004:~# ip route
default via 192.168.10.200 dev eth0 proto static
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.123
[root@centos7-3 ~]# hostnamectl set-hostname route
[root@route ~]# cdnet
[root@route network-scripts]# vim ifcfg-eth0
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.200
PREFIX=24
:wq
[root@route network-scripts]# cp ifcfg-eth0 ifcfg-eth1
[root@route network-scripts]# vim ifcfg-eth1
DEVICE=eth1
NAME=eth1
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.200
PREFIX=24
:wq
[root@route network-scripts]# reboot
[root@route ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6c:54:0a brd ff:ff:ff:ff:ff:ff
inet 10.0.0.200/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6c:540a/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6c:54:14 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.200/24 brd 192.168.10.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6c:5414/64 scope link
valid_lft forever preferred_lft forever
[root@route ~]# sysctl -a |grep ip_forward
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.eth1.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_use_pmtu = 0
[root@route ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
:wq
[root@route ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@rs1 network-scripts]# vim ifcfg-eth0
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.7
PREFIX=24
GATEWAY=10.0.0.200
DNS1=223.5.5.5
DNS2=180.76.76.76
:wq
[root@rs1 network-scripts]# reboot
[root@rs2 network-scripts]# vim ifcfg-eth0
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.17
PREFIX=24
GATEWAY=10.0.0.200
DNS1=223.5.5.5
DNS2=180.76.76.76
:wq
[root@rs2 network-scripts]# reboot
[root@centos8 ~]# dnf -y install ipvsadm
[root@centos8 network-scripts]# vim ifcfg-eth0
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.8
PREFIX=24
GATEWAY=10.0.0.200
DNS1=223.5.5.5
DNS2=180.76.76.76
:wq
[root@centos8 network-scripts]# reboot
[root@rs2 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 [::]:8080 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*
[root@rs2 ~]# vim /etc/httpd/conf/httpd.conf
Listen 80
:wq
[root@rs2 ~]# systemctl restart httpd
[root@rs2 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*
[root@centos8 ~]# hostnamectl set-hostname lvs
[root@lvs ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100
[root@lvs ~]# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=64 time=0.366 ms
^C
--- 10.0.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.366/0.366/0.366/0.000 ms
[root@lvs ~]# ping 10.0.0.17
PING 10.0.0.17 (10.0.0.17) 56(84) bytes of data.
64 bytes from 10.0.0.17: icmp_seq=1 ttl=64 time=1.13 ms
^C
--- 10.0.0.17 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.126/1.126/1.126/0.000 ms
root@ubuntu2004:~# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=63 time=0.776 ms
^C
--- 10.0.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.776/0.776/0.776/0.000 ms
[root@rs1 ~]# ping 10.0.0.8
PING 10.0.0.8 (10.0.0.8) 56(84) bytes of data.
64 bytes from 10.0.0.8: icmp_seq=1 ttl=64 time=0.280 ms
^C
--- 10.0.0.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.280/0.280/0.280/0.000 ms
[root@rs2 ~]# ping 10.0.0.8
PING 10.0.0.8 (10.0.0.8) 56(84) bytes of data.
64 bytes from 10.0.0.8: icmp_seq=1 ttl=64 time=0.276 ms
^C
--- 10.0.0.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.276/0.276/0.276/0.000 ms
root@ubuntu2004:~# ping 10.0.0.17
PING 10.0.0.17 (10.0.0.17) 56(84) bytes of data.
64 bytes from 10.0.0.17: icmp_seq=1 ttl=63 time=0.756 ms
64 bytes from 10.0.0.17: icmp_seq=2 ttl=63 time=1.32 ms
^C
--- 10.0.0.17 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1031ms
rtt min/avg/max/mdev = 0.756/1.036/1.316/0.280 ms
[root@rs1 ~]# ping 192.168.10.123
PING 192.168.10.123 (192.168.10.123) 56(84) bytes of data.
64 bytes from 192.168.10.123: icmp_seq=1 ttl=63 time=0.947 ms
64 bytes from 192.168.10.123: icmp_seq=2 ttl=63 time=0.647 ms
^C
--- 192.168.10.123 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.647/0.797/0.947/0.150 ms
[root@rs1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@rs1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@rs1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@rs1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@rs2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@rs2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@rs2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@rs2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@rs1 ~]# ifconfig lo:1 10.0.0.100/32
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.100/0 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link
valid_lft forever preferred_lft forever
[root@rs2 network-scripts]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.100/0 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:be:39:4d brd ff:ff:ff:ff:ff:ff
inet 10.0.0.17/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:febe:394d/64 scope link
valid_lft forever preferred_lft forever
[root@lvs ~]# ifconfig lo:1 10.0.0.100/32
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.100/0 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:3f:19:17 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe3f:1917/64 scope link
valid_lft forever preferred_lft forever
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lvs ~]# ipvsadm -A -t 10.0.0.100:80 -s rr
[root@lvs ~]# ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.7 -g
[root@lvs ~]# ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.17 -g
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.100:80 rr
-> 10.0.0.7:80 Route 1 0 0
-> 10.0.0.17:80 Route 1 0 0
root@ubuntu2004:~# curl 10.0.0.100
10.0.0.17
rs2
root@ubuntu2004:~# curl 10.0.0.100
10.0.0.7
rs1
root@ub
[root@lvs ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100
[root@lvs ~]# ip route del default via 10.0.0.200 dev eth0 proto static metric 100
[root@lvs ~]# ip route
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100
root@ubuntu2004:~# while true;do curl 10.0.0.100;sleep 1;done
[root@lvs ~]# ip route add default via 10.0.0.222 dev eth0 proto static metric 100
[root@lvs ~]# ip route
default via 10.0.0.222 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100
root@ubuntu2004:~# while true;do curl 10.0.0.100;sleep 1;done
10.0.0.7
rs1
10.0.0.17
rs2
LVS上必须加网关，可以随便加网关只要跟route是一个网段即可
lo网卡子网掩码必须是32，不然不能通讯
五.LVS-DR模式多网段案例
[root@route ~]# ip a a 172.16.0.200/24 dev eth0 label eth0:1
[root@route ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6c:54:0a brd ff:ff:ff:ff:ff:ff
inet 10.0.0.200/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.16.0.200/24 scope global eth0:1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6c:540a/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6c:54:14 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.200/24 brd 192.168.10.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6c:5414/64 scope link
valid_lft forever preferred_lft forever
[root@route ~]# ip route
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.200 metric 100
172.16.0.0/24 dev eth0 proto kernel scope link src 172.16.0.200
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.200 metric 101
[root@lvs ~]# reboot
[root@rs1 ~]# reboot
[root@rs2 ~]# reboot
[root@rs1 ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.7 metric 100
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link
valid_lft forever preferred_lft forever
[root@rs2 ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.17 metric 100
[root@rs2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:be:39:4d brd ff:ff:ff:ff:ff:ff
inet 10.0.0.17/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:febe:394d/64 scope link
valid_lft forever preferred_lft forever
[root@rs1 ~]# vim lvs_dr_rs.sh
#!/bin/bash
#Author:wangxiaochun
#Date:2017-08-13
vip=172.16.0.100
mask='255.255.255.255'
dev=lo:1
#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
:wq
[root@rs1 ~]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.16.0.100/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link
valid_lft forever preferred_lft forever
[root@rs2 ~]# vim lvs_dr_rs.sh
#!/bin/bash
#Author:wangxiaochun
#Date:2017-08-13
vip=172.16.0.100
mask='255.255.255.255'
dev=lo:1
#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
:wq
[root@rs1 ~]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.16.0.100/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link
valid_lft forever preferred_lft forever
[root@rs2 ~]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@rs2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.16.0.100/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:be:39:4d brd ff:ff:ff:ff:ff:ff
inet 10.0.0.17/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:febe:394d/64 scope link
valid_lft forever preferred_lft forever
[root@lvs ~]# vim lvs_dr_vs.sh
#!/bin/bash
#Author:wangxiaochun
#Date:2017-08-13
vip='172.16.0.100'
iface='lo:1'
mask='255.255.255.255'
port='80'
rs1='10.0.0.7'
rs2='10.0.0.17'
scheduler='wrr'
type='-g'
rpm -q ipvsadm &> /dev/null || yum -y install ipvsadm &> /dev/null
case $1 in
start)
ifconfig $iface $vip netmask $mask #broadcast $vip up
iptables -F
ipvsadm -A -t ${vip}:${port} -s $scheduler
ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
echo "The VS Server is Ready!"
;;
stop)
ipvsadm -C
ifconfig $iface down
echo "The VS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
:wq
[root@lvs ~]# bash lvs_dr_vs.sh start
The VS Server is Ready!
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.0.100:80 wrr
-> 10.0.0.7:80 Route 1 0 0
-> 10.0.0.17:80 Route 1 0 2
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.16.0.100/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:3f:19:17 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe3f:1917/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu2004:~# while true;do curl 172.16.0.100;sleep 1;done
10.0.0.17
rs2
10.0.0.7
rs1
