K8s ipvs mode kube-proxy
IPVS vs. IPTABLES
IPVS模式在Kubernetes 1.8中被引入，在1.9中进入beta测试。 IPTABLES模式在1.1版本中被添加进来，在1.2开始就变成了默认的操作模式。IPVS 和 IPTABLES都是基于netfilter实现的。
IPVS与IPTABLES有以下几点不同的地方:
-
IPVS为大规模集群提供更好的可扩展性和性能。
-
IPVS比IPTABLES支持更复杂的负载均衡算法 (least load, least connections, locality, weighted, etc.)。
-
IPVS支持服务器健康检查和重连等等。
配置kube-proxy ipvs模式
k8s版本：1.10
默认情况下，Kube-proxy依然使用iptables来实现Service到POD之间的负载均衡。
[root@node01 ~]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR httpd-svc ClusterIP 10.110.73.171 <none> 8080/TCP 1d run=httpd kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d <none> [root@node01 ~]# iptables-save
.
.
. -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE -A KUBE-SEP-6E7XQMQ4RAYOWTTM -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ -A KUBE-SEP-6E7XQMQ4RAYOWTTM -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.3:53 -A KUBE-SEP-7SDFQQ3CPAUK3SXM -s 10.244.1.2/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-7SDFQQ3CPAUK3SXM -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.1.2:80 -A KUBE-SEP-FXPWU3QR6KR3AMXC -s 172.16.65.180/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ -A KUBE-SEP-FXPWU3QR6KR3AMXC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-FXPWU3QR6KR3AMXC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.65.180:6443 -A KUBE-SEP-IT2ZTR26TO4XFPTO -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ -A KUBE-SEP-IT2ZTR26TO4XFPTO -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.2:53 -A KUBE-SEP-TOSIKZVXYIAJSFHP -s 10.244.2.2/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-TOSIKZVXYIAJSFHP -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.2.2:80 -A KUBE-SEP-UO4LFRT7VSL3X4EB -s 10.244.0.4/32 -m comment --comment "default/httpd-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-UO4LFRT7VSL3X4EB -p tcp -m comment --comment "default/httpd-svc:" -m tcp -j DNAT --to-destination 10.244.0.4:80 -A KUBE-SEP-YIL6JZP7A3QYXJU2 -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ -A KUBE-SEP-YIL6JZP7A3QYXJU2 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.2:53 -A KUBE-SEP-ZXMNUKOKXUTL2MK2 -s 10.244.0.3/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ -A KUBE-SEP-ZXMNUKOKXUTL2MK2 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.3:53 -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4 -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.110.73.171/32 -p tcp -m comment --comment "default/httpd-svc: cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.110.73.171/32 -p tcp -m comment --comment "default/httpd-svc: cluster IP" -m tcp --dport 8080 -j KUBE-SVC-RL3JAE4GN7VOGDGP -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS -A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-IT2ZTR26TO4XFPTO -A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZXMNUKOKXUTL2MK2 -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-FXPWU3QR6KR3AMXC --mask 255.255.255.255 --rsource -j KUBE-SEP-FXPWU3QR6KR3AMXC -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-FXPWU3QR6KR3AMXC -A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-UO4LFRT7VSL3X4EB -A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7SDFQQ3CPAUK3SXM -A KUBE-SVC-RL3JAE4GN7VOGDGP -m comment --comment "default/httpd-svc:" -j KUBE-SEP-TOSIKZVXYIAJSFHP -A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YIL6JZP7A3QYXJU2 -A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-6E7XQMQ4RAYOWTTM .
.
.
第一步，在内核中加载ip_vs模块：
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack_ipv4" for kernel_module in \${ipvs_modules}; do /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1 if [ $? -eq 0 ]; then /sbin/modprobe \${kernel_module} fi done EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
输出结果应该为：
[root@node01 ~]# lsmod | grep ip_vs ip_vs_ftp 13079 0 ip_vs_sed 12519 0 ip_vs_nq 12516 0 ip_vs_sh 12688 0 ip_vs_dh 12688 0 ip_vs_lblcr 12922 0 ip_vs_lblc 12819 0 ip_vs_wrr 12697 0 ip_vs_rr 12600 3 ip_vs_wlc 12519 0 ip_vs_lc 12516 0 nf_nat 26787 3 ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4 ip_vs 141092 27 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_lblcr,ip_vs_lblc nf_conntrack 133387 7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4 libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
第二步，安装ipvs管理工具ipvsadm
yum install -y ipvsadm
第三步，修改集群配置文件
在使用kubeadm init --config config.yaml初始化集群前，修改集群配置文件
Kubernetes v1.8 v1.9 kind: MasterConfiguration apiVersion: kubeadm.k8s.io/v1alpha1 ... kubeProxy: config: featureGates: SupportIPVSProxyMode=true mode: ipvs ... Kubernetes v1.10 kind: MasterConfiguration apiVersion: kubeadm.k8s.io/v1alpha1 ... kubeProxy: config: featureGates: SupportIPVSProxyMode: true mode: ipvs ...
第四步，结果验证
[root@node01 ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.96.0.1:443 rr persistent 10800 -> 172.16.65.180:6443 Masq 1 3 0 TCP 10.96.0.10:53 rr -> 10.244.0.2:53 Masq 1 0 0 -> 10.244.0.3:53 Masq 1 0 0 UDP 10.96.0.10:53 rr -> 10.244.0.2:53 Masq 1 0 0 -> 10.244.0.3:53 Masq 1 0 0
第五步，测试
创建http-svc service
apiVersion: apps/v1beta1 kind: Deployment metadata: name: httpd spec: replicas: 3 template: metadata: labels: run: httpd spec: containers: - name: httpd image: httpd ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: httpd-svc spec: selector: run: httpd
type: NodePort ports: - protocol: TCP port: 8080 targetPort: 80
nodePort: 30001
查看效果
[root@node01 yaml-store]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE httpd-749bf8c6f4-htzgl 1/1 Running 0 11m 10.244.2.2 node03 httpd-749bf8c6f4-vqwff 1/1 Running 0 11m 10.244.1.2 node02 httpd-749bf8c6f4-wkv9w 1/1 Running 0 11m 10.244.0.4 node01
[root@node01 yaml-store]# kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR httpd-svc ClusterIP 10.99.129.90 <none> 8080:30001/TCP 11m run=httpd kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 58m <none>
[root@node01 yaml-store]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.65.181:30001 rr -> 10.244.0.4:80 Masq 1 0 0 -> 10.244.1.2:80 Masq 1 0 0 -> 10.244.2.2:80 Masq 1 0 0 TCP 172.17.0.1:30001 rr -> 10.244.0.4:80 Masq 1 0 0 -> 10.244.1.2:80 Masq 1 0 0 -> 10.244.2.2:80 Masq 1 0 0 TCP 10.96.0.1:443 rr persistent 10800 -> 172.16.65.180:6443 Masq 1 3 0 TCP 10.96.0.10:53 rr -> 10.244.0.2:53 Masq 1 0 0 -> 10.244.0.3:53 Masq 1 0 0 TCP 10.99.129.90:8080 rr -> 10.244.0.4:80 Masq 1 0 0 -> 10.244.1.2:80 Masq 1 0 0 -> 10.244.2.2:80 Masq 1 0 0 TCP 10.244.0.0:30001 rr -> 10.244.0.4:80 Masq 1 0 0 -> 10.244.1.2:80 Masq 1 0 0 -> 10.244.2.2:80 Masq 1 0 0 TCP 10.244.0.1:30001 rr -> 10.244.0.4:80 Masq 1 0 0 -> 10.244.1.2:80 Masq 1 0 0 -> 10.244.2.2:80 Masq 1 0 0 TCP 127.0.0.1:30001 rr -> 10.244.0.4:80 Masq 1 0 0 -> 10.244.1.2:80 Masq 1 0 0 -> 10.244.2.2:80 Masq 1 0 0 UDP 10.96.0.10:53 rr -> 10.244.0.2:53 Masq 1 0 0 -> 10.244.0.3:53 Masq 1 0 0
参考文章：
https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs
https://www.kubernetes.org.cn/3025.html
