第十章 Kubernetes的CNI网络插件--flannel
1、简介
1.1前言
Kubernetes设计了网络模型，但却将它的实现讲给了网络插件，CNI网络插件最重要的功能就是实现Pod资源能够跨主机通信
常见的CNI网络插件如下：
Flannel；
Cacliao；
Canal；
Contiv；
OpenContrail；
NSX-T；
Kube-router。
1.2 Flannel的三种网络模型
- host-gw模型：所有的node ip必须在同一个物理网关设备下才能使用，他的原理就是：给宿主机添加一个静态路由
- Vxlan模型：当宿主机不在同一个网段下（不适用同一个网关），就可以使用此模型，相当于在每台宿主机上添加了一个虚拟网络设备，通过一个虚拟隧道进行通信。
- 直接路由模型：当node不在同一个物理网关，走vxlan模型，当在同一个路由下，走host-gw模型
2、集群规划
部署以10.4.7.21为例，10.4.7.22部署类似
主机名 角色 IP
hdss7-21 Flannel 10.4.7.21
hdss7-22 Flannel 10.4.7.22
3、下载软件，解压，做软连接
下载地址：https://github.com/flannel-io/flannel/
[root@hdss7-21 ~]# cd /opt/src/
[root@hdss7-21 src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@hdss7-21 src]# mkdir /opt/flannel-v0.11.0
[root@hdss7-21 src]# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0
[root@hdss7-21 src]# ln -s /opt/flannel-v0.11.0 /opt/flannel
4、最终目录结构
[root@hdss7-21 src]# cd /opt/flannel
[root@hdss7-21 flannel]# ls
flanneld mk-docker-opts.sh README.md
5、拷贝client证书
作为etcd的客户端
[root@hdss7-21 flannel]# mkdir cert
[root@hdss7-21 flannel]# cd cert/
root@hdss7-21 cert]# scp hdss7-200:/opt/certs/ca.pem .
root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client.pem .
root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client-key.pem .
root@hdss7-21 cert]# ll
总用量 12
-rw-r--r-- 1 root root 1346 6月 13 21:55 ca.pem
-rw------- 1 root root 1679 6月 13 21:56 client-key.pem
-rw-r--r-- 1 root root 1363 6月 13 21:55 client.pem
6、创建配置
注意：flannel集群各主机的配置不同，部署其他节点时注意修改，10.4.7.22上FLANNEL_SUBNET=172.7.22.1/24
[root@hdss7-21 cert]# cd ..
[root@hdss7-21 flannel]# vim subnet.env
FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
7、创建启动脚本
注意IP（--public-ip=）与网络接口（--iface=）修改
[root@hdss7-21 flannel]# vim flanneld.sh
#!/bin/sh
./flanneld \
--public-ip=10.4.7.21 \
--etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--etcd-keyfile=./cert/client-key.pem \
--etcd-certfile=./cert/client.pem \
--etcd-cafile=./cert/ca.pem \
--iface=ens33 \
--subnet-file=./subnet.env \
--healthz-port=2401
8、检查配置、权限、创建日志目录
[root@hdss7-21 flannel]# chmod +x flanneld.sh
[root@hdss7-21 flannel]# mkdir -p /data/logs/flanneld
9、操作etcd，增加host-gw
在etcd集群中，可以在12,21,22上任意一台上操作
[root@hdss7-21 flannel]# cd /opt/etcd
注意网络地址
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
查看
[root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
查看一下etcd集群信息
[root@hdss7-21 etcd]# ./etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
10、创建supervisor配置
[root@hdss7-21 etcd]# vim /etc/supervisord.d/flannel.ini
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/flannel ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
11、启动服务并检查
[root@hdss7-21 etcd]# supervisorctl update
flanneld-7-21: added process group
[root@hdss7-21 etcd]# tail -100f /data/logs/flanneld/flanneld.stdout.log
I0613 22:58:53.387404 33642 main.go:527] Using interface with name ens33 and address 10.4.7.21
I0613 22:58:53.387528 33642 main.go:540] Using 10.4.7.21 as external address
2021-06-13 22:58:53.388681 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
I0613 22:58:53.388761 33642 main.go:244] Created subnet manager: Etcd Local Manager with Previous Subnet: 172.7.21.0/24
I0613 22:58:53.388772 33642 main.go:247] Installing signal handlers
I0613 22:58:53.396039 33642 main.go:587] Start healthz server on 0.0.0.0:2401
I0613 22:58:53.418509 33642 main.go:386] Found network config - Backend type: host-gw
I0613 22:58:53.432160 33642 local_manager.go:201] Found previously leased subnet (172.7.21.0/24), reusing
I0613 22:58:53.440005 33642 local_manager.go:220] Allocated lease (172.7.21.0/24) to current node (10.4.7.21)
I0613 22:58:53.440627 33642 main.go:317] Wrote subnet file to ./subnet.env
I0613 22:58:53.440644 33642 main.go:321] Running backend.
I0613 22:58:53.441000 33642 route_network.go:53] Watching for new subnet leases
I0613 22:58:53.452691 33642 main.go:429] Waiting for 22h59m59.980128025s to renew lease
I0613 22:58:53.452920 33642 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0613 22:58:53.452975 33642 iptables.go:167] Deleting iptables rule: -s 172.7.0.0/16 -j ACCEPT
I0613 22:58:53.464074 33642 iptables.go:167] Deleting iptables rule: -d 172.7.0.0/16 -j ACCEPT
I0613 22:58:53.471984 33642 iptables.go:155] Adding iptables rule: -s 172.7.0.0/16 -j ACCEPT
I0613 22:58:53.497924 33642 iptables.go:155] Adding iptables rule: -d 172.7.0.0/16 -j ACCEPT
12、部署集群其他节点，检查所有集群服务
上面的3-11（etcd不需要操作了）步骤在10.4.7.22上操作部署；
等22上的flannel服务起来之后检查路由，发现增加了到22容器网段通过10.4.7.22主机的路由（22也有到21的网络），这就是flannel做的事情（但是注意，物理主机需要在同一个网关下才能使用host-gw网络模型）
查看路由
[root@hdss7-21 etcd]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 ens33
10.4.7.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
172.7.21.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.7.22.0 10.4.7.22 255.255.255.0 UG 0 0 0 ens33
22上有道21的网关路由：
[root@hdss7-21 etcd]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 ens33
10.4.7.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
172.7.21.0 10.4.7.21 255.255.255.0 UG 0 0 0 ens33
172.7.22.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
13、再次验证集群，POD网路互通
在21上访问22上的容器，已经可以互通了
[root@hdss7-21 etcd]# curl 172.7.22.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
在22上访问21上的容器
[root@hdss7-22 flannel]# curl 172.7.21.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
14、在各个运算节点上优化iptables规则
为什么要做iptables规则优化：默认主机上是做好了snat的规则转换，就是容器访问到容器时，在目标容器内显示的源地址是主机地址，而不是源容器地址，如下所示，在10.4.7.21上的172.7.21.2容器访问172.7.22.2容器时，在172.7.22.2内查看到源地址的是10.4.7.21，这样就不知道到底是哪个具体容器进行访问的了，我们要做的就是把容器到容器的iptables的snat转换规则去掉。
以下操作在10.4.7.21上操作
14.1 容器添加附加curl功能
修改使用的镜像为curl镜像（之前在第四章部署harbor的时候创建过此镜像）
[root@hdss7-21 ~]# vim /opt/kubernetes/conf/nginx-ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ds
spec:
template:
metadata:
labels:
app: nginx-ds
spec:
containers:
- name: my-nginx
image: harbor.od.com/public/nginx:curl
ports:
- containerPort: 80
[root@hdss7-21 ~]# kubectl apply -f /opt/kubernetes/conf/nginx-ds.yaml
daemonset.extensions/nginx-ds configured
用新的镜像替换原来的容器
~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ds-spdgm 1/1 Running 2 3d 172.7.21.2 hdss7-21.host.com <none> <none>
nginx-ds-sx7hn 1/1 Running 0 111s 172.7.22.2 hdss7-22.host.com <none> <none>
原来的容器被kill后会根据期望自动生成新的容器
[root@hdss7-21 ~]# kubectl delete pod nginx-ds-spdgm
pod "nginx-ds-spdgm" deleted
[root@hdss7-21 ~]# kubectl delete pod nginx-ds-sx7hn
pod "nginx-ds-sx7hn" deleted
[root@hdss7-21 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ds-97b8r 1/1 Running 0 12s 172.7.21.2 hdss7-21.host.com <none> <none>
nginx-ds-ncpk8 1/1 Running 0 4s 172.7.22.2 hdss7-22.host.com <none> <none>
14.2 优化前访问
进入172.7.21.2容器对172.7.22.2容器进行访问，可能会出现容器无法启动的情况（由于容器使用的dns是192.168.0.2，后面部署好dns就没问题了），那么换成原来的容器，参看本章16节的内容，使容器可以ping通外网后安装curl工具就行了
查看当前的pod信息
~]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ds-nqb57 1/1 Running 0 22m 172.7.21.2 hdss7-21.host.com <none> <none>
nginx-ds-pmgm2 1/1 Running 0 22m 172.7.22.2 hdss7-22.host.com <none> <none>
进入172.7.21.2容器访问172.7.22.2容器，查看172.7.22.2的日志，会发现源地址是172.7.21.2的宿主机即10.4.7.21访问的；
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-nqb57 bash
root@nginx-ds-nqb57:/# curl 172.7.22.2
[root@hdss7-21 ~]# kubectl logs -f nginx-ds-pmgm2
10.4.7.21 - - [17/Jun/2021:13:22:10 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
14.3 开始优化iptables规则
以21为例，22网络地址不同
[root@hdss7-21 ~]# yum install iptables-services -y
[root@hdss7-21 ~]# systemctl start iptables && systemctl enable iptables
[root@hdss7-21 ~]# iptables-save | grep -i postrouting
:POSTROUTING ACCEPT [1:63]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables-save | grep -i postrouting
:POSTROUTING ACCEPT [53:3186]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
将iptables自带的拒绝规则删除
[root@hdss7-21 ~]# iptables-save | grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
15、各运算节点保存iptables规则
15.1 保存规则
[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables
[root@hdss7-21 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ 确定 ]
再次尝试容器之间访问
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-nqb57 bash
root@nginx-ds-nqb57:/# curl 172.7.22.2
[root@hdss7-21 ~]# kubectl logs -f nginx-ds-pmgm2
10.4.7.21 - - [17/Jun/2021:13:16:55 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
10.4.7.21 - - [17/Jun/2021:13:22:10 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
172.7.21.2 - - [17/Jun/2021:13:33:07 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
15.2 重启docker服务
注意：修改后会影响到docker原本的iptables链的规则，所以需要重启docker服务
docker重启后之前删除的nat规则会再次生效，filter不会
[root@hdss7-21 ~]# systemctl restart docker
[root@hdss7-21 ~]# iptables-save |grep -i postrouting|grep docker0
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
可以用iptables-restore重新应用iptables规则,也可以直接再删
[root@hdss7-21 ~]# iptables-restore /etc/sysconfig/iptables
[root@hdss7-21 ~]# iptables-save |grep -i postrouting|grep docker0
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
16、报错排查
当你发现使用kubectl启动的容器无法ping通外网时，尝试使用本地的docker通过同一个镜像创建一个容器，如果此容器正常访问外网，那么可以说明宿主机没有问题，docker没有问题（如果有问题，网上的一些说法是重启一下docker），那么可能就是kubectl的配置有问题，比较一下两种容器的dns，发现不一样；
Docker启动的容器如下：
~]# docker exec -it f2be1d19ff19 bash
root@f2be1d19ff19:/# cat /etc/resolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.4.7.11
root@f2be1d19ff19:/# ping baidu.com
PING baidu.com (39.156.69.79): 48 data bytes
56 bytes from 39.156.69.79: icmp_seq=0 ttl=53 time=10.950 ms
使用kubectl启动的容器如下：
~]# kubectl exec -it nginx-ds-gmhm7 bash
root@nginx-ds-gmhm7:/# cat /etc/resolv.conf
nameserver 192.168.0.2
search default.svc.cluster.local svc.cluster.local cluster.local host.com
options ndots:5
root@nginx-ds-gmhm7:/# ping baidu.com
ping: unknown host
发现，容器的ip地址和dns都不在同一个网段，将dns修改成正常docker启动的dns就行了
root@nginx-ds-gmhm7:/# echo "nameserver 10.4.7.11" > /etc/resolv.conf
root@nginx-ds-gmhm7:/# ping baidu.com
PING baidu.com (220.181.38.148): 48 data bytes
56 bytes from 220.181.38.148: icmp_seq=0 ttl=49 time=15.832 ms
root@nginx-ds-gmhm7:/# tee /etc/apt/sources.list << EOF
deb http://mirrors.163.com/debian/ jessie main non-free contrib
deb http://mirrors.163.com/debian/ jessie-updates main non-free contrib
EOF
root@nginx-ds-gmhm7:/# apt-get update && apt-get install curl -y
尝试curl百度首页
root@cc8ae2b47946:/# curl -k https://www.baidu.com
关于容器dns为192.168.0.2，无法访问外网，这个问题在后面部署好dns后就可以解决了
