Keepalived + LVS-NAT 实现高可用四层 TCP/UDP 负载均衡器

2019-01-04 19:55 云物互联阅读(2297) 评论(0) 编辑收藏举报

前文列表

在 LVS1/2 安装 Keepalived & LVS

[root@control01 ~]# yum install -y keepalived ipvsadm

[root@control01 ~]# keepalived --version
Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

[root@control01 ~]# ipvsadm --version
ipvsadm v1.27 2008/5/15 (compiled with popt and IPVS v1.2.1)

Keepalived + LVS-NAT 实现 TCP 负载均衡

官方文档：LVS NAT + Keepalived HOWTO

IP 规划

Client：192.168.1.100/24
LVS1：
- 外部网卡 192.168.1.110/24
- 内部网卡 10.0.0.103/24
LVS2：
- 外部网卡 192.168.1.111/24
- 内部网卡 10.0.0.104/24
VS 外部 VIP：192.168.1.112
VS 内部 DIP：10.0.0.105
RS1：
- IP 10.0.0.101/24
- 网关 10.0.0.105/24
RS2：
- IP 10.0.0.102/24
- 网关 10.0.0.105/24

网络架构参考

在这里插入图片描述

LVS1 配置

启动 ipvsadm：

[root@localhost ~]# touch /etc/sysconfig/ipvsadm
[root@localhost ~]# systemctl start ipvsadm.service

启动 keepalived：

[root@localhost ~]# cat /etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D -d"

[root@localhost ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   lvs_id LVS_01
}

vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}

vrrp_instance VI_1 {
        state MASTER
        interface eno16777736
        lvs_sync_daemon_inteface eno16777736
        virtual_router_id 51
        priority 150
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass 1111
        }

        virtual_ipaddress {
                192.168.1.112
        }
}

vrrp_instance VI_GATEWAY {
        state MASTER
        interface eno33554960
        lvs_sync_daemon_inteface eno33554960
        virtual_router_id 52
        priority 150
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                10.0.0.105
        }
}

virtual_server 192.168.1.112 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol TCP

    real_server 10.0.0.101 80 {
        weight 1
    }
    real_server 10.0.0.102 80 {
        weight 1
    }
}

[root@localhost ~]# systemctl start keepalived

Keepalived 的启动日志：

[root@localhost ~]# journalctl -f -u keepalived
-- Logs begin at Sun 2019-01-06 07:05:29 EST. --
Jan 06 09:57:02 localhost.localdomain systemd[1]: Starting LVS and VRRP High Availability Monitor...
Jan 06 09:57:02 localhost.localdomain Keepalived[18040]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Jan 06 09:57:02 localhost.localdomain Keepalived[18040]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 06 09:57:02 localhost.localdomain Keepalived[18041]: Starting Healthcheck child process, pid=18042
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Initializing ipvs
Jan 06 09:57:02 localhost.localdomain Keepalived[18041]: Starting VRRP child process, pid=18043
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Unknown keyword 'lvs_id'
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Unknown keyword 'nat_mask'
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Registering Kernel netlink reflector
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Registering Kernel netlink command channel
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Registering gratuitous ARP shared channel
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Unknown keyword 'lvs_id'
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Unknown keyword 'lvs_sync_daemon_inteface'
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Unknown keyword 'lvs_sync_daemon_inteface'
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_GATEWAY) removing protocol VIPs.
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ------< Global definitions >------
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Router ID = localhost
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Default interface = eth0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: LVS flush = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP IPv4 mcast group = 224.0.0.18
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP IPv6 mcast group = ff02::12
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP refresh timer = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP refresh repeat = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP lower priority delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP lower priority repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Send advert after receive lower priority advert = true
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Send advert after receive higher priority advert = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP interval = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous NA interval = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP default protocol version = 2
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Iptables input chain = INPUT
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Using ipsets = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ipset IPv4 address set = keepalived
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ipset IPv6 address set = keepalived6
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ipset IPv6 address,iface set = keepalived_if6
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP check unicast_src = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP skip check advert addresses = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP strict mode = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP process priority = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP don't swap = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Checker process priority = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Checker don't swap = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: SNMP keepalived disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: SNMP checker disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: SNMP RFCv2 disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: SNMP RFCv3 disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: SNMP traps disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: SNMP socket = default (unix:/var/agentx/master)
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Network namespace = (default)
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Script security disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Default script uid:gid 0:0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ------< VRRP Topology >------
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP Instance = VI_1
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Using VRRPv2
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Want State = MASTER
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Running on device = eno16777736
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Skip checking advert IP addresses = no
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Enforcing strict VRRP compliance = no
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Using src_ip = 192.168.1.110
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP refresh timer = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP refresh repeat = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP lower priority delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP lower priority repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Send advert after receive lower priority advert = true
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Send advert after receive higher priority advert = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Virtual Router ID = 51
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Priority = 150
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Advert interval = 1 sec
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Accept enabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Promote_secondaries disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Authentication type = SIMPLE_PASSWORD
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Password = 1111
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Virtual IP = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: ------< Global definitions >------
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Router ID = localhost
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: 192.168.1.112/32 dev eno16777736 scope global
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Default interface = eth0
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: LVS flush = false
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP IPv4 mcast group = 224.0.0.18
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP IPv6 mcast group = ff02::12
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP refresh timer = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP refresh repeat = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP lower priority delay = 4294
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP lower priority repeat = -1
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Send advert after receive lower priority advert = true
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Send advert after receive higher priority advert = false
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous ARP interval = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Gratuitous NA interval = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP Instance = VI_GATEWAY
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP default protocol version = 2
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Using VRRPv2
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Iptables input chain = INPUT
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Want State = MASTER
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Using ipsets = true
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Running on device = eno33554960
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: ipset IPv4 address set = keepalived
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Skip checking advert IP addresses = no
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: ipset IPv6 address set = keepalived6
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Enforcing strict VRRP compliance = no
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: ipset IPv6 address,iface set = keepalived_if6
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Using src_ip = 10.0.0.103
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP check unicast_src = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP skip check advert addresses = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP strict mode = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP refresh timer = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP process priority = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP refresh repeat = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VRRP don't swap = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP lower priority delay = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Checker process priority = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Gratuitous ARP lower priority repeat = 5
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Checker don't swap = false
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Send advert after receive lower priority advert = true
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: SNMP keepalived disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Send advert after receive higher priority advert = false
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: SNMP checker disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Virtual Router ID = 52
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: SNMP RFCv2 disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Priority = 150
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: SNMP RFCv3 disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Advert interval = 1 sec
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: SNMP traps disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Accept enabled
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: SNMP socket = default (unix:/var/agentx/master)
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Promote_secondaries disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Network namespace = (default)
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Authentication type = SIMPLE_PASSWORD
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Script security disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Password = example
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Default script uid:gid 0:0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Virtual IP = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: ------< SSL definitions >------
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: 10.0.0.105/32 dev eno33554960 scope global
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Using autogen SSL context
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ------< VRRP Sync groups >------
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: ------< LVS Topology >------
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP Sync Group = VG1, BACKUP
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: System is compiled with LVS v1.2.1
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: monitor = VI_1
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: VIP = 192.168.1.112, VPORT = 80
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: monitor = VI_GATEWAY
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Address family = inet
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ------< NIC >------
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: delay_loop = 6, lb_algo = rr
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Name = eno16777736
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: Hashed = disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: index = 2
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: flag-1 = disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: IPv4 address = 192.168.1.110
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: flag-2 = disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: IPv6 address = fe80::20c:29ff:fe27:d53b
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: flag-3 = disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: MAC = 00:0c:29:27:d5:3b
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: One packet scheduling = disabled
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: is UP
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: protocol = TCP
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: is RUNNING
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: alpha is OFF, omega is OFF
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: MTU = 1500
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: quorum = 1, hysteresis = 0
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: HW Type = ETHERNET
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: lb_kind = NAT
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: ------< NIC >------
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: RIP = 10.0.0.101, RPORT = 80, WEIGHT = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Name = eno33554960
Jan 06 09:57:02 localhost.localdomain Keepalived_healthcheckers[18042]: RIP = 10.0.0.102, RPORT = 80, WEIGHT = 1
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: index = 3
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: IPv4 address = 10.0.0.103
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: IPv6 address = fe80::20c:29ff:fe27:d545
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: MAC = 00:0c:29:27:d5:45
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: is UP
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: is RUNNING
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: MTU = 1500
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: HW Type = ETHERNET
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: Using LinkWatch kernel netlink reflector...
Jan 06 09:57:02 localhost.localdomain systemd[1]: Started LVS and VRRP High Availability Monitor.
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jan 06 09:57:02 localhost.localdomain Keepalived_vrrp[18043]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(12,13)]
Jan 06 09:57:03 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_GATEWAY) Transition to MASTER STATE
Jan 06 09:57:03 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_GATEWAY) Entering MASTER STATE
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_GATEWAY) setting protocol VIPs.
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_GATEWAY) Sending/queueing gratuitous ARPs on eno33554960 for 10.0.0.105
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Group(VG1) Syncing instances to MASTER state
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eno16777736 for 192.168.1.112
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:04 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_GATEWAY) Sending/queueing gratuitous ARPs on eno33554960 for 10.0.0.105
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno33554960 for 10.0.0.105
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eno16777736 for 192.168.1.112
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112
Jan 06 09:57:09 localhost.localdomain Keepalived_vrrp[18043]: Sending gratuitous ARP on eno16777736 for 192.168.1.112

查看 VIP 和 DIP：

[root@localhost ~]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:27:d5:3b brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.110/24 brd 192.168.1.255 scope global dynamic eno16777736
       valid_lft 6646sec preferred_lft 6646sec
    inet 192.168.1.112/32 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe27:d53b/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever
3: eno33554960: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:27:d5:45 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.103/24 brd 10.0.0.255 scope global eno33554960
       valid_lft forever preferred_lft forever
    inet 10.0.0.105/32 scope global eno33554960
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe27:d545/64 scope link
       valid_lft forever preferred_lft forever

查看 ipvs 规则：

[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.112:80 rr
  -> 10.0.0.101:80                Masq    1      0          0
  -> 10.0.0.102:80                Masq    1      0          0

NOTE：LVS1 上并不会真的开启 80 端口，只是 80 只是一个 VPORT。

[root@localhost ~]# netstat -lpntu | grep 80

开启路由转发功能：

[root@localhost ~]# cat /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.ip_forward = 1

[root@localhost ~]# sysctl -p
net.ipv4.ip_forward = 1

清空防火墙规则：

[root@localhost ~]# iptables -F -t filter
[root@localhost ~]# iptables -F -t raw
[root@localhost ~]# iptables -F -t mangle
[root@localhost ~]# iptables -F -t nat

LVS2 配置

配置 LVS2 和配置 LVS1 的步骤基本一致，但 Keepalived 的配置文件有些许变动。

[root@localhost ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   lvs_id LVS_01
}

vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}

vrrp_instance VI_1 {
        state BACKUP
        interface eno16777736
        lvs_sync_daemon_inteface eno16777736
        virtual_router_id 51
        priority 140
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass 1111
        }

        virtual_ipaddress {
                192.168.1.112
        }
}

vrrp_instance VI_GATEWAY {
        state BACKUP
        interface eno33554960
        lvs_sync_daemon_inteface eno33554960
        virtual_router_id 52
        priority 150
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                10.0.0.105
        }
}

virtual_server 192.168.1.112 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol TCP

    real_server 10.0.0.101 80 {
        weight 1
    }
    real_server 10.0.0.102 80 {
        weight 1
    }
}

配置 RS1

网卡的 Gateway 指向 DIP：

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno33554960
HWADDR=00:0C:29:15:40:15
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eno33554960
UUID=be63d7a3-f7eb-4204-9c1d-cecb2e857d0b
ONBOOT=yes
IPADDR=10.0.0.101
GATEWAY=10.0.0.105
NETMASK=255.255.255.0
DNS1=114.114.114.114

清空防火墙规则：

[root@localhost ~]# iptables -F -t filter
[root@localhost ~]# iptables -F -t raw
[root@localhost ~]# iptables -F -t mangle
[root@localhost ~]# iptables -F -t nat

安装 TCP 80 的 httpd 服务：

[root@localhost ~]# yum install -y httpd

[root@localhost ~]# cat /var/www/html/index.html
<html>
  <body>
    <h1>RS1</h1>
  </body>
</html>

[root@localhost ~]# systemctl start httpd

[root@localhost ~]# netstat -lpntu | grep 80
tcp6       0      0 :::80                   :::*                    LISTEN      18227/httpd

配置 RS2

配置 RS2 与 RS1 的步骤基本一致，只是 httpd 的 index.html 有些许改变：

[root@localhost ~]# cat /var/www/html/index.html
<html>
  <body>
    <h1>RS2</h1>
  </body>
</html>

验证

在客户端 curl VIP:VPort 会轮询的访问 RS1 和 RS2。

[root@localhost ~]# curl 192.168.1.112
<html>
  <body>
    <h1>RS2</h1>
  </body>
</html>
[root@localhost ~]# curl 192.168.1.112
<html>
  <body>
    <h1>RS1</h1>
  </body>
</html>

在 MASTER 查看连接表：

[root@localhost ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state       source             virtual            destination
TCP 01:27  TIME_WAIT   192.168.1.100:52034 192.168.1.112:80   10.0.0.101:80

在 BACKUP 查看连接表：

[root@localhost ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state       source             virtual            destination

Failover：关闭 MASTER 电源之后还可以继续通过客户端访问 curl VIP:VPort，查看发现 VIP 漂移到了 BACKUP，再次查看 BACKUP 的 ipvs 连接表：

[root@localhost ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state       source             virtual            destination
TCP 01:57  TIME_WAIT   192.168.1.100:52115 192.168.1.112:80   10.0.0.101:80

TS

开始为了方便操作 RS1/RS2 所以也为这两天机器添加了 192.168.1.0/24 的网卡，但这样做的话 VIP 无法生效，后来卸载掉即可，暂不清楚什么原因。

Keepalived + LVS-NAT 实现 UDP 负载均衡

配置 RS1/2

启用 nc udp 服务器，接收外部传输过来的文件：

[root@localhost ~]# yum install -y nc

[root@localhost ~]# nc -ul 9999 > file.txt

[root@localhost ~]# netstat -lpntu | grep 9999
udp        0      0 0.0.0.0:9999            0.0.0.0:*                           2618/nc
udp6       0      0 :::9999                 :::*                                2618/nc

配置 LVS1/2

配置 Keepalived，添加 UDP 协议负载均衡 virtual_server：

[root@localhost ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   lvs_id LVS_01
}

vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}

vrrp_instance VI_1 {
        state MASTER
        interface eno16777736
        lvs_sync_daemon_inteface eno16777736
        virtual_router_id 51
        priority 150
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass 1111
        }

        virtual_ipaddress {
                192.168.1.112
        }
}

vrrp_instance VI_GATEWAY {
        state MASTER
        interface eno33554960
        lvs_sync_daemon_inteface eno33554960
        virtual_router_id 52
        priority 150
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass example
        }
        virtual_ipaddress {
                10.0.0.105
        }
}

virtual_server 192.168.1.112 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol TCP

    real_server 10.0.0.101 80 {
        weight 1
    }
    real_server 10.0.0.102 80 {
        weight 1
    }
}

virtual_server 192.168.1.112 9999 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol UDP

    real_server 10.0.0.101 9999 {
        weight 1
    }
    real_server 10.0.0.102 9999 {
        weight 1
    }
}

[root@localhost ~]# systemctl restart keepalived

[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.112:80 rr
  -> 10.0.0.101:80                Masq    1      0          0
  -> 10.0.0.102:80                Masq    1      0          0
UDP  192.168.1.112:9999 rr
  -> 10.0.0.101:9999              Masq    1      0          0
  -> 10.0.0.102:9999              Masq    1      0          0

NOTE：LVS1/2 的配置大同小异，只是 vrrp_instance 的角色和权重有所区别而已。

验证

在客户端上准备两个文件：

╭─mickeyfan@localhost  ~/test
╰─$ cat 1.txt                                                                                                                                                                    1 ↵
11111111111
╭─mickeyfan@localhost  ~/test
╰─$ cat 2.txt
22222222222

执行文件传输：

╭─mickeyfan@localhost  ~/test
╰─$ nc -u -w 1 192.168.1.112 9999 < 1.txt                                                                                                                                        1 ↵
╭─mickeyfan@localhost  ~/test
╰─$ nc -u -w 1 192.168.1.112 9999 < 2.txt

可以看见这两个文件分别被 RS1 和 RS2 的 nc udp 服务端接收。

[root@localhost ~]# nc -ul 9999 > file.txt

Ncat: Connection refused.
[root@localhost ~]# cat file.txt
22222222222

[root@localhost ~]# nc -ul 9999 > file.txt

Ncat: Connection refused.
[root@localhost ~]# cat file.txt
11111111111

查看 ipvs 转发表：

[root@localhost ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state       source             virtual            destination
UDP 04:20  UDP         192.168.1.100:65136 192.168.1.112:9999 10.0.0.101:9999
UDP 04:19  UDP         192.168.1.100:51930 192.168.1.112:9999 10.0.0.102:9999

总结

使用 Keepalived + LVS-NAT 模式需要注意以下几点：

LVS 服务器应该具有两张网卡，并且通过 Keepalived 分别配置 VIP 和 DIP。VIP 是给外部客户端访问的，DIP 是给内部后端服务器访问的。
LVS 服务器应该开启路由转发功能。
RS 服务器的网关应该指向 DIP。
ipvs 的规则应该是 VIP 转发到后端真实服务器，而不是 DIP 转发到后端真实服务器。

刷新页面返回顶部

云物互联云计算、云原生、5G 网络、边缘计算。

Keepalived + LVS-NAT 实现高可用四层 TCP/UDP 负载均衡器

目录

文章目录

前文列表

在 LVS1/2 安装 Keepalived & LVS

Keepalived + LVS-NAT 实现 TCP 负载均衡

IP 规划

网络架构参考

LVS1 配置

LVS2 配置

配置 RS1

配置 RS2

验证

TS

Keepalived + LVS-NAT 实现 UDP 负载均衡

配置 RS1/2

配置 LVS1/2

验证

总结

About

云物互联 云计算、云原生、5G 网络、边缘计算。

Keepalived + LVS-NAT 实现高可用四层 TCP/UDP 负载均衡器

目录

文章目录

前文列表

在 LVS1/2 安装 Keepalived & LVS

Keepalived + LVS-NAT 实现 TCP 负载均衡

IP 规划

网络架构参考

LVS1 配置

LVS2 配置

配置 RS1

配置 RS2

验证

TS

Keepalived + LVS-NAT 实现 UDP 负载均衡

配置 RS1/2

配置 LVS1/2

验证

总结

About

云物互联云计算、云原生、5G 网络、边缘计算。