lvs的nat和dr模式混合用
机器部署信息
lvs ：
10.0.0.200 vip
10.0.0.19 外网IP ，
172.168.1.19 内网IP
dr rs:
10.0.0.200 vip
10.0.0.18 rip
nat rs:
172.168.1.17 rip
客户端：
10.0.0.14 cip
lvs机器：
ip addr add 10.0.0.200/24 dev ens33:0
IP：
[root@mcw09 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:f0:dd:56 brd ff:ff:ff:ff:ff:ff inet 10.0.0.19/24 brd 10.0.0.255 scope global ens33 valid_lft forever preferred_lft forever inet 10.0.0.200/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::495b:ff7:d185:f95d/64 scope link valid_lft forever preferred_lft forever inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:f0:dd:60 brd ff:ff:ff:ff:ff:ff inet 172.168.1.19/24 brd 172.168.1.255 scope global ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever [root@mcw09 ~]#
路由没有啥变动，
[root@mcw09 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.254 0.0.0.0 UG 100 0 0 ens34 0.0.0.0 10.0.0.254 0.0.0.0 UG 101 0 0 ens33 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw09 ~]#
dr的rs和lvs用同一个网段，nat的转发和lvs可以是两个网段
[root@mcw09 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.0.0.200:80 rr -> 172.168.1.17:80 Masq 1 0 0 -> 10.0.0.18:80 Route 1 0 0 [root@mcw09 ~]#
dr机器：
route add -host 10.0.0.200 dev lo
[root@mcw08 ~]# cat /etc/sysctl.conf net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2
lo添加了VIP
[root@mcw08 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 10.0.0.200/32 scope global lo:0 valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:26:33:3f brd ff:ff:ff:ff:ff:ff inet 10.0.0.18/24 brd 10.0.0.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::f32c:166d:40de:8f2e/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::495b:ff7:d185:f95d/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:26:33:49 brd ff:ff:ff:ff:ff:ff inet 172.168.1.18/24 brd 172.168.1.255 scope global ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 15: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100 link/none inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0 valid_lft forever preferred_lft forever inet6 fe80::923d:6caf:c22:c8a5/64 scope link flags 800 valid_lft forever preferred_lft forever [root@mcw08 ~]#
nat机器：
[root@mcw07 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5d:df:62 brd ff:ff:ff:ff:ff:ff 3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5d:df:6c brd ff:ff:ff:ff:ff:ff inet 172.168.1.17/24 brd 172.168.1.255 scope global ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever [root@mcw07 ~]#
只开启了一个内网IP，并且默认网关指向dip。这里的dip和rip是同一个网段的。不在同一个网段的不清楚咋弄
[root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 0 0 0 ens34 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]#
测试：
mcw04上测试，访问lvs上VIP 10.0.0.200,从而访问到两个rs,一个rs是nat，一个是dr
添加其他网络测试
当把nat的另外一个网卡起起来之后
[root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 0 0 0 ens34 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]# ifup ens33 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/45) [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 0 0 0 ens34 0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 ens33 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]#
nat的这个rs机器就不能访问到了
[root@mcw04 ~]# curl 10.0.0.200:80 curl: (7) Failed connect to 10.0.0.200:80; Connection timed out [root@mcw04 ~]# curl 10.0.0.200:80 rs1 mcw08 ^_^ 10.0.0.18 [root@mcw04 ~]# curl 10.0.0.200:80 curl: (7) Failed connect to 10.0.0.200:80; Connection timed out [root@mcw04 ~]# curl 10.0.0.200:80 rs1 mcw08 ^_^ 10.0.0.18 [root@mcw04 ~]#
删掉一条nat的rs的默认路由之后还是无法访问到nat的
[root@mcw07 ~]# [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 0 0 0 ens34 0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 ens33 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]# ip route del default via 10.0.0.254 dev ens33 [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 0 0 0 ens34 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]#
[root@mcw04 ~]# curl 10.0.0.200:80 rs1 mcw08 ^_^ 10.0.0.18 [root@mcw04 ~]# curl 10.0.0.200:80 curl: (7) Failed connect to 10.0.0.200:80; Connection timed out [root@mcw04 ~]#
因为内网dip做路由
内网网卡卡做默认路由时，如何去通外网。需要添加外网网段指向外网网卡(lvs nat rs中有多网卡处理案例)
dip和rip是同一个内网，因为将nat模型的lvs的dip作为默认网关后，这样ens33的能通223.5.5.5的网卡，现在不通了
[root@mcw07 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5d:df:62 brd ff:ff:ff:ff:ff:ff inet 10.0.0.17/24 brd 10.0.0.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::f32c:166d:40de:8f2e/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::495b:ff7:d185:f95d/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5d:df:6c brd ff:ff:ff:ff:ff:ff inet 172.168.1.17/24 brd 172.168.1.255 scope global ens34 valid_lft forever preferred_lft forever inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed valid_lft forever preferred_lft forever inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed valid_lft forever preferred_lft forever [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]# [root@mcw07 ~]# [root@mcw07 ~]# [root@mcw07 ~]# ping 223.5.5.5 PING 223.5.5.5 (223.5.5.5) 56(84) bytes of data. From 172.168.1.19 icmp_seq=1 Redirect Host(New nexthop: 172.168.1.254) From 172.168.1.19: icmp_seq=1 Redirect Host(New nexthop: 172.168.1.254) ^C --- 223.5.5.5 ping statistics --- 3 packets transmitted, 0 received, +1 errors, 100% packet loss, time 2003ms [root@mcw07 ~]#
添加走223.5.5.5的，还是用ens33网卡，这个网卡之前默认网关是10.0.0.254.现在直接指定走这个网段的，都指定网关和网卡。这样就能重新通223.5.5.5这个网络了。
ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33
[root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]# ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33 [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 223.0.0.0 10.0.0.254 255.0.0.0 UG 0 0 0 ens33 [root@mcw07 ~]# ping 223.5.5.5 PING 223.5.5.5 (223.5.5.5) 56(84) bytes of data. 64 bytes from 223.5.5.5: icmp_seq=1 ttl=128 time=9.00 ms 64 bytes from 223.5.5.5: icmp_seq=2 ttl=128 time=7.96 ms ^C --- 223.5.5.5 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 7.969/8.485/9.002/0.524 ms [root@mcw07 ~]#
将下面跟ens33网卡的路由删除。最后只保留ens34的两条路由。这样情况下，两个网卡内的网段，发现网络都是互通的，也就是路由上没有这个网卡的配置，这个网卡相关的网段好像也是可以直接通的。再添加一个走223.0.0.0网段的路由，走ens33网卡接口和它对应的网关，这样就能通223.5.5.5了。
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 [root@mcw07 ~]# ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33 [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 223.0.0.0 10.0.0.254 255.0.0.0 UG 0 0 0 ens33 [root@mcw07 ~]# ip route del 10.0.0.0/24 dev ens33 [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 223.0.0.0 10.0.0.254 255.0.0.0 UG 0 0 0 ens33 [root@mcw07 ~]# ping 223.5.5.5 PING 223.5.5.5 (223.5.5.5) 56(84) bytes of data. 64 bytes from 223.5.5.5: icmp_seq=1 ttl=128 time=25.8 ms 64 bytes from 223.5.5.5: icmp_seq=2 ttl=128 time=8.13 ms ^C --- 223.5.5.5 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 8.133/16.975/25.818/8.843 ms [root@mcw07 ~]# ping 10.0.0.18 PING 10.0.0.18 (10.0.0.18) 56(84) bytes of data. 64 bytes from 10.0.0.18: icmp_seq=1 ttl=64 time=6.44 ms 64 bytes from 10.0.0.18: icmp_seq=2 ttl=64 time=0.690 ms ^C --- 10.0.0.18 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.690/3.569/6.449/2.880 ms [root@mcw07 ~]# ping 10.0.0.19 PING 10.0.0.19 (10.0.0.19) 56(84) bytes of data. 64 bytes from 10.0.0.19: icmp_seq=1 ttl=64 time=0.919 ms ^C --- 10.0.0.19 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.919/0.919/0.919/0.000 ms [root@mcw07 ~]#
现在下面ens33的跟10.0.0.0/24 10.0.0.254 相关的路由都已经删除，只保留了ens34的172.168.1.0/24 172.168.1.254这个两条路由，然后新增一个走向223.5.5.5这个外网IP时，走ens33网卡接口的路由，指定ens33的网关，这样223.5.5.5在mcw07上由不通变为通。并且不会影响lvs nat 下rs的功能。也就是lvs那里正常访问到mcw07.。如果加上ens33的路由，比如只加上10.0.0.0/24 ，网关是0.0.0.0时就无法让lvs nat正常响应数据了。
[root@mcw07 ~]# [root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 223.0.0.0 10.0.0.254 255.0.0.0 UG 0 0 0 ens33 [root@mcw07 ~]#
客户端正常响应数据，返回nat模型的数据，也就是mcw07的数据。不知道是否在某种情况下，可以添加ens33网卡原本有的路由条目，不过即使没加，但是10.0.0.0/24网段的IP也是通的，但是其它网段想通，需要指定走ens33，也就是我们这里这个案例的这种配置，不然可能是走的内网网卡ens34,这个本来就是默认不通外网的。也就是你想要通的网段，如果现在的默认网关ens34不通，但是用ens33通的话，需要添加该网段路由，指向ens33，这样它就知道怎么走，可以通网了，不然就是走的默认的ens34，这个不通的了
[root@mcw04 ~]# curl 10.0.0.200:80 rs1 mcw08 ^_^ 10.0.0.18 [root@mcw04 ~]# curl 10.0.0.200:80 rs2 mcw07 ^_^ 10.0.0.17 [root@mcw04 ~]#
下面是lvs规则
[root@mcw09 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.0.0.200:80 rr -> 172.168.1.17:80 Masq 1 0 0 -> 10.0.0.18:80 Route 1 0 0 [root@mcw09 ~]#
给lvs nat 的rs添加arp抑制，不影响nat的正常
[root@mcw07 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.168.1.19 0.0.0.0 UG 100 0 0 ens34 172.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens34 223.0.0.0 10.0.0.254 255.0.0.0 UG 0 0 0 ens33 [root@mcw07 ~]# vim /etc/sysctl.conf [root@mcw07 ~]# tail -5 /etc/sysctl.conf net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2 [root@mcw07 ~]# sysctl -p
mcw07这个nat模型的rs,正常被访问到
[root@mcw04 ~]# curl 10.0.0.200:80 rs1 mcw08 ^_^ 10.0.0.18 [root@mcw04 ~]# curl 10.0.0.200:80 rs2 mcw07 ^_^ 10.0.0.17 [root@mcw04 ~]#
