Kubernetes IPVS和IPTABLES

个人名片:
对人间的热爱与歌颂,可抵岁月冗长🌞
Github👨🏻‍💻:念舒_C.ying
CSDN主页✏️:念舒_C.ying
个人博客🌏 :念舒_C.ying

什么是IPVS

IPVS(IP Virtual Server,IP虚拟服务器)实现了传输层的负载平衡,通常称为4 LAN(四层局域网)交换,是Linux内核的一部分。

IPVS在主机上运行,在真实服务器集群前面充当负载平衡器。IPVS可以将基于 TCP 和 UDP 的服务请求定向到真实服务器上。

IPVS vs IPTABLES

IPVS 模式在 Kubernetes v1.8中引入,在v1.9中成为测试版,在v1.11中成为GA。
IPTABLES模式是在v1.1版本中加入的,从v1.2版本开始成为默认的操作模式。

IPVS和IPTABLES都是基于netfilter的。IPVS模式和IPTABLES模式的区别如下:

  1. IPVS为大型集群提供更好的可扩展性和性能。

  2. IPVS比IPTABLES支持更复杂的负载平衡算法(最小负载、最小连接、定位、加权等)。

  3. IPVS 支持服务器健康检查和连接重试等。

IPVS 对 IPTABLES 的依赖

IPVS代理使用IPTABLES做数据包过滤SNAT伪装。具体来说,IPVS代理将使用ipset来存储需要DROP或做伪装的流量的源地址或目的地址,以确保无论我们有多少服务,IPTABLES规则的数量不变。

下面是IPVS代理服务器使用的ipset集的表格。

set namemembersusage
KUBE-CLUSTER-IPAll service IP + portMark-Masq for cases that masquerade-all=true or clusterCIDR specified
KUBE-LOOP-BACKAll service IP + port + IPmasquerade for solving hairpin purpose
KUBE-EXTERNAL-IPservice external IP + portmasquerade for packages to external IPs
KUBE-LOAD-BALANCERload balancer ingress IP + portmasquerade for packages to load balancer type service
KUBE-LOAD-BALANCER-LOCALLB ingress IP + port with externalTrafficPolicy=localaccept packages to load balancer with externalTrafficPolicy=local
KUBE-LOAD-BALANCER-FWload balancer ingress IP + port with loadBalancerSourceRangespackage filter for load balancer with loadBalancerSourceRanges specified
KUBE-LOAD-BALANCER-SOURCE-CIDRload balancer ingress IP + port + source CIDRpackage filter for load balancer with loadBalancerSourceRanges specified
KUBE-NODE-PORT-TCPnodeport type service TCP portmasquerade for packets to nodePort(TCP)
KUBE-NODE-PORT-LOCAL-TCPnodeport type service TCP port with externalTrafficPolicy=localaccept packages to nodeport service with externalTrafficPolicy=local
KUBE-NODE-PORT-UDPnodeport type service UDP portmasquerade for packets to nodePort(UDP)
KUBE-NODE-PORT-LOCAL-UDPnodeport type service UDP port with externalTrafficPolicy=localaccept packages to nodeport service with externalTrafficPolicy=local

在以下情况下,IPVS 代理将依赖 IPTABLES。

1. kube-proxy以–masquerade-all=true启动

如果kube-proxy以--masquerade-all=true启动,IPVS代理将伪装所有访问服务集群IP的流量,这与IPTABLES代理的行为相同。假设kube-proxy指定了标志--masquerade-all=true,那么IPVS代理安装的IPTABLES应该如下所示:

# iptables -t nat -nL

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */

Chain KUBE-MARK-MASQ (2 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst

2. 在kube-proxy启动时指定集群CIDR

如果kube-proxy以--cluster-cidr=<cidr>启动,IPVS代理将伪装访问服务集群IP的非集群流量,其行为与IPTABLES代理相同。假设kube-proxy提供的集群cidr是10.244.16.0/24,那么IPVS代理安装的IPTABLES应该如下所示。

# iptables -t nat -nL

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */

Chain KUBE-MARK-MASQ (3 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  -- !10.244.16.0/24       0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst

3. loadBalancer类型的服务

对于loadBalancer类型的服务,IPVS代理将安装IPTABLES与ipset KUBE-LOAD-BALANCER匹配。特别是当服务的LoadBalancerSourceRanges被指定或指定externalTrafficPolicy=local时,IPVS代理将创建ipset集KUBE-LOAD-BALANCER-LOCAL/KUBE-LOAD-BALANCER-FW/KUBE-LOAD-BALANCER-SOURCE-CIDR并相应地安装IPTABLES,它应该看起来像下面所示。

# iptables -t nat -nL

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */

Chain KUBE-FIREWALL (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER-SOURCE-CIDR dst,dst,src
KUBE-MARK-DROP  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-LOAD-BALANCER (1 references)
target     prot opt source               destination
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER-FW dst,dst
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER-LOCAL dst,dst
KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-MARK-DROP (1 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (2 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-LOAD-BALANCER  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER dst,dst
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER dst,dst

4. NodePort类型的服务

对于NodePort类型的服务,IPVS代理将安装IPTABLES与ipset KUBE-NODE-PORT-TCP/KUBE-NODE-PORT-UDP的匹配。当指定externalTrafficPolicy=local时,IPVS代理将创建ipset集KUBE-NODE-PORT-LOCAL-TCP/KUBE-NODE-PORT-LOCAL-UDP并相应地安装IPTABLES,这应该是如下所示的:

假设服务的TCP类型为nodePort。

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */

Chain KUBE-MARK-MASQ (2 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODE-PORT (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-NODE-PORT-LOCAL-TCP dst
KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-NODE-PORT  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-NODE-PORT-TCP dst

5. 指定externalIP的服务

对于指定了外部IP的服务,IPVS代理将安装IPTABLES与ipset KUBE-EXTERNAL-IP匹配,假设我们有指定了外部IP的服务,IPTABLES规则应该如下所示:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */

Chain KUBE-MARK-MASQ (2 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOOP-BACK dst,dst,src

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-EXTERNAL-IP dst,dst
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-EXTERNAL-IP dst,dst PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-EXTERNAL-IP dst,dst ADDRTYPE match dst-type LOCAL

Kubernetes使用IPVS

安装IPVS

CentOS

yum install ipset ipvsadm -y

Ubuntu

apt-get install ipset ipvsadm -y

设置

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

期待下次的分享,别忘了三连支持博主呀~
我是 念舒_C.ying ,期待你的关注~💪💪💪

posted @ 2022-11-18 22:35  偶尔栖息  阅读(782)  评论(1编辑  收藏  举报