分析kube-proxy的iptables规则
NodePort service
创建一个mysql的NodePort服务，对应两个pod实例，rc和service的配置如下：
1、rc配置
apiVersion: v1 kind: ReplicationController metadata: name: wordpress-mysql spec: replicas: 2 selector: name: wordpress-mysql template: metadata: labels: name: wordpress-mysql spec: containers: - name: wordpress-mysql image: 172.16.114.201/library/mysql:v1 ports: - containerPort: 3306 volumeMounts: - name: "wordpress-mysql-data" mountPath: "/var/lib/mysql" env: - name: MYSQL_PASS value: "123456" - name: ON_CREATE_DB value: "wordpress" volumes: - name: "wordpress-mysql-data" hostPath: path: "/root/wordpress-mysql/data"
2、service配置
apiVersion: v1 kind: Service metadata: name: wordpress-mysql spec: ports: - port: 3306 targetPort: 3306 nodePort: 30010 protocol: TCP type: NodePort selector: name: wordpress-mysql
3、创建的service情况
Name: wordpress-mysql Namespace: default Labels: <none> Selector: name=wordpress-mysql Type: NodePort IP: 10.254.67.85 Port: <unset> 3306/TCP NodePort: <unset> 30010/TCP Endpoints: 10.0.3.2:3306,10.0.45.6:3306 Session Affinity: None No events.
4、kube-proxy占用端口情况
[root@test-209 log]# netstat -anp | grep kube-proxy tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 10165/kube-proxy tcp 0 0 172.16.114.209:46010 172.16.114.208:8080 ESTABLISHED 10165/kube-proxy tcp 0 0 172.16.114.209:46014 172.16.114.208:8080 ESTABLISHED 10165/kube-proxy tcp 0 0 172.16.114.209:46012 172.16.114.208:8080 ESTABLISHED 10165/kube-proxy tcp6 0 0 :::30010 :::* LISTEN 10165/kube-proxy unix 2 [ ] DGRAM 36395 10165/kube-proxy unix 3 [ ] STREAM CONNECTED 36403 10165/kube-proxy
5、对应的iptables规则
iptables -S -t nat | grep mysql -A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport 30010 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport 30010 -j KUBE-SVC-GJ6HULPZPPQIKMS7 -A KUBE-SEP-7KXQQUXVSZ2LFV44 -s 10.0.45.6/32 -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ -A KUBE-SEP-7KXQQUXVSZ2LFV44 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.45.6:3306 -A KUBE-SEP-J7SZJXRP24HRFT23 -s 10.0.3.2/32 -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ -A KUBE-SEP-J7SZJXRP24HRFT23 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.3.2:3306 -A KUBE-SERVICES -d 10.254.67.85/32 -p tcp -m comment --comment "default/wordpress-mysql: cluster IP" -m tcp --dport 3306 -j KUBE-SVC-GJ6HULPZPPQIKMS7 -A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-J7SZJXRP24HRFT23 -A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -j KUBE-SEP-7KXQQUXVSZ2LFV44
从以上信息看出，kube-proxy为mysql服务在node节点上单独起了一个端口30010，在iptables的规则中，目的端口30010被指向KUBE-SVC-GJ6HULPZPPQIKMS7，KUBE-SVC-GJ6HULPZPPQIKMS7又被指向KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44（他两各50%的几率），KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44定义了DNAT转换规则，将访问重定向到10.0.45.6:3306和10.0.3.2:3306这两个endpoint。因此，当外部访问30010端口时，根据iptables的规则会将该消息分发给10.0.45.6:3306和10.0.3.2:3306这两个地址（分发的几率是各50%）
ClusterIP service
创建一个zookeeper的ClusterIP服务，rc和service的配置如下：
1、rc配置
apiVersion: v1 kind: ReplicationController metadata: name: zookeeper1 spec: replicas: 1 selector: name: zookeeper1 template: metadata: labels: name: zookeeper1 spec: containers: - name: zookeeper1 image: 10.10.30.166/public/zookeeper:v1 ports: - containerPort: 2181 - containerPort: 2888 - containerPort: 3888 env: - name: ZOOKEEPER_ID value: "1" - name: ZOOKEEPER_SERVER_1 value: "zookeeper1" - name: ZOOKEEPER_SERVER_2 value: "zookeeper2" - name: ZOOKEEPER_SERVER_3 value: "zookeeper3"
2、service配置
apiVersion: v1 kind: Service metadata: name: zookeeper1 spec: ports: - port: 2181 targetPort: 2181 protocol: TCP name: "1" - port: 2888 targetPort: 2888 protocol: TCP name: "2" - port: 3888 targetPort: 3888 protocol: TCP name: "3" type: ClusterIP selector: name: zookeeper1
3、创建service情况
Name: zookeeper1 Namespace: default Labels: <none> Selector: name=zookeeper1 Type: ClusterIP IP: 10.254.181.6 Port: 1 2181/TCP Endpoints: 10.0.45.4:2181 Port: 2 2888/TCP Endpoints: 10.0.45.4:2888 Port: 3 3888/TCP Endpoints: 10.0.45.4:3888 Session Affinity: None No events.
4、iptables规则
iptables -S -t nat | grep zookeeper1 -A KUBE-SEP-BZJZKIUQRVYJVMQB -s 10.0.45.4/32 -m comment --comment "default/zookeeper1:3" -j KUBE-MARK-MASQ -A KUBE-SEP-BZJZKIUQRVYJVMQB -p tcp -m comment --comment "default/zookeeper1:3" -m tcp -j DNAT --to-destination 10.0.45.4:3888 -A KUBE-SEP-C3J2QHMJ3LTD3GR7 -s 10.0.45.4/32 -m comment --comment "default/zookeeper1:2" -j KUBE-MARK-MASQ -A KUBE-SEP-C3J2QHMJ3LTD3GR7 -p tcp -m comment --comment "default/zookeeper1:2" -m tcp -j DNAT --to-destination 10.0.45.4:2888 -A KUBE-SEP-RZ4H7H2HFI3XFCXZ -s 10.0.45.4/32 -m comment --comment "default/zookeeper1:1" -j KUBE-MARK-MASQ -A KUBE-SEP-RZ4H7H2HFI3XFCXZ -p tcp -m comment --comment "default/zookeeper1:1" -m tcp -j DNAT --to-destination 10.0.45.4:2181 -A KUBE-SERVICES -d 10.254.181.6/32 -p tcp -m comment --comment "default/zookeeper1:1 cluster IP" -m tcp --dport 2181 -j KUBE-SVC-HHEJUKXW5P7DV7BX -A KUBE-SERVICES -d 10.254.181.6/32 -p tcp -m comment --comment "default/zookeeper1:2 cluster IP" -m tcp --dport 2888 -j KUBE-SVC-2SVOYTXLXAXVV7L3 -A KUBE-SERVICES -d 10.254.181.6/32 -p tcp -m comment --comment "default/zookeeper1:3 cluster IP" -m tcp --dport 3888 -j KUBE-SVC-KAVJ7GO67HRSOAM3 -A KUBE-SVC-2SVOYTXLXAXVV7L3 -m comment --comment "default/zookeeper1:2" -j KUBE-SEP-C3J2QHMJ3LTD3GR7 -A KUBE-SVC-HHEJUKXW5P7DV7BX -m comment --comment "default/zookeeper1:1" -j KUBE-SEP-RZ4H7H2HFI3XFCXZ -A KUBE-SVC-KAVJ7GO67HRSOAM3 -m comment --comment "default/zookeeper1:3" -j KUBE-SEP-BZJZKIUQRVYJVMQB
从iptables的规则来看，对目的ip是10.254.181.6，端口是2181、2888或者3888的消息，规则指向了KUBE-SVC-HHEJUKXW5P7DV7BX、KUBE-SVC-2SVOYTXLXAXVV7L3、KUBE-SVC-KAVJ7GO67HRSOAM3，他们三又分别指向了KUBE-SEP-C3J2QHMJ3LTD3GR7、KUBE-SEP-RZ4H7H2HFI3XFCXZ、KUBE-SEP-BZJZKIUQRVYJVMQB，这三条规则定义了DNAT转换规则，将访问重定向到了10.0.45.4:3888、10.0.45.4:2888、10.0.45.4:2181
