3.kubernetes的CNI网络插件-Flannel
目录
1.1.K8S的CNI网络插件-Flannel
集群规划
| 主机名 | 角色 | ip |
| HDSS7-21.host.com | flannel | 10.4.7.21 |
| HDSS7-22.host.com | flannel | 10.4.7.22 |
下载软件、解压、软链接
官网下载地址:https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
HDSS7-21.host.com上:
[root@hdss7-21 ~]# cd /opt/src/ [root@hdss7-21 ~]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz [root@hdss7-21 ~]# mkdir /opt/flannel-v0.11.0 [root@hdss7-21 ~]# tar zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0/ [root@hdss7-21 ~]# ln -s /opt/flannel-v0.11.0 /opt/flannel [root@hdss7-21 ~]# ll /opt/|grep flannel lrwxrwxrwx 1 root root 16 Jun 9 10:00 flannel -> flannel-v0.11.0/ drwxr-xr-x 3 root root 4096 Jun 9 10:04 flannel-v0.11.0
最终目录结构
[root@hdss7-21 opt]# tree -L 2 . ├── containerd │ ├── bin │ └── lib ├── etcd -> etcd-v3.1.20 ├── etcd-v3.1.20 │ ├── certs │ ├── Documentation │ ├── etcd │ ├── etcdctl │ ├── etcd-server-startup.sh │ ├── README-etcdctl.md │ ├── README.md │ └── READMEv2-etcdctl.md ├── flannel -> flannel-v0.11.0/ ├── flannel-v0.11.0 │ ├── certs │ ├── flanneld │ ├── mk-docker-opts.sh │ └── README.md ├── kubernetes -> kubernetes-v1.15.2/ ├── kubernetes-v1.15.2 │ ├── addons │ ├── LICENSES │ └── server └── src ├── etcd-v3.1.20-linux-amd64.tar.gz ├── flannel-v0.11.0-linux-amd64.tar.gz └── kubernetes-server-linux-amd64-v1.15.2.tar.gz
拷贝证书
各运算节点上:
/opt/flannel/certs
[root@hdss7-21 certs]# scp hdss7-200:/opt/certs/ca.pem . [root@hdss7-21 certs]# scp hdss7-200:/opt/certs/client.pem . [root@hdss7-21 certs]# scp hdss7-200:/opt/certs/client-key.pem .
注:其他运算节点也需要ca.pem client.pem client-key.pem
创建配置
[root@hdss7-21 flannel]# vi subnet.env FLANNEL_NETWORK=172.7.0.0/16 FLANNEL_SUBNET=172.7.21.1/24 FLANNEL_MTU=1500 FLANNEL_IPMASQ=false
注:其他运算节点需将FLANNEL_SUBNET改成对应的网段，比如:10.4.7.22配置FLANNEL_SUBNET=172.7.22.1/24
创建启动脚本
[root@hdss7-21 flannel]# vim flanneld.sh #!/bin/sh ./flanneld \ --public-ip=10.4.7.21 \ --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \ --etcd-keyfile=./certs/client-key.pem \ --etcd-certfile=./certs/client.pem \ --etcd-cafile=./certs/ca.pem \ --iface=eth0 \ --subnet-file=./subnet.env \ --healthz-port=2401
注:flannel集群各主机的启动脚本略有不同，部署其他节点注意修改，比如10.4.7.22 --public-ip=10.4.7.22
检查配置、权限、创建日志目录
[root@hdss7-21 flannel]# chmod +x /opt/flannel/flanneld.sh [root@hdss7-21 flannel]# mkdir -p /data/logs/flanneld
操作etcd,增加host-gw（适用于运算节点在同一个二层模型,三选一）
HDSS7-21.host.com上:
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}' {"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}} [root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config {"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
创建supervisor配置
HDSS7-21.host.com上:
[root@hdss7-21 ~]# vi /etc/supervisord.d/flannel.ini [program:flanneld-7-21] command=/opt/flannel/flanneld.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/flannel ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/flanneld/flanneld.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
注:hdss7-22.host.com需要修改成[program:flanneld-7-22]
启动服务并检查
[root@hdss7-21 flannel]# supervisorctl update [root@hdss7-21 flannel]# supervisorctl status
再次验证集群，pod之间网络互通
[root@hdss7-21 ~]# ping 172.7.22.2 PING 172.7.22.2 (172.7.22.2) 56(84) bytes of data. 64 bytes from 172.7.22.2: icmp_seq=1 ttl=63 time=8.50 ms 64 bytes from 172.7.22.2: icmp_seq=2 ttl=63 time=2.58 ms 64 bytes from 172.7.22.2: icmp_seq=3 ttl=63 time=4.40 ms 64 bytes from 172.7.22.2: icmp_seq=4 ttl=63 time=0.353 ms 64 bytes from 172.7.22.2: icmp_seq=5 ttl=63 time=0.420 ms
flannel vxlan模型（适用于运算节点不在同一个二层，三选一）
1.停掉flanneld服务 [root@hdss7-21 ~]# supervisor stop flanneld-7-21 [root@hdss7-22 ~]# supervisor stop flanneld-7-22 2.删除host-gw模型创建的路由 [root@hdss7-21 ~]# route del -net 172.7.21.0/24 gw 10.4.7.21 [root@hdss7-22 ~]# route del -net 172.7.22.0/24 gw 10.4.7.22 3.修改etcd [root@hdss7-21 ~]# cd /opt/etcd [root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config [root@hdss7-21 etcd]# ./etcdctl rm /coreos.com/network/config [root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "VxLAN"}}' [root@hdss7-21 etcd]# supervisor start flanneld-7-21 [root@hdss7-22 ~]# supervisor start flanneld-7-22
flannel 直接路由模型（自动识别是否在同一个二层模型，如果是同一个二层模型走host-gw，如果在同一个二层模型走vxlan模型，三选一）
1.停掉flanneld服务 [root@hdss7-21 ~]# supervisor stop flanneld-7-21 [root@hdss7-22 ~]# supervisor stop flanneld-7-22 2.修改etcd [root@hdss7-21 ~]# cd /opt/etcd [root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config [root@hdss7-21 etcd]# ./etcdctl rm /coreos.com/network/config [root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "VxLAN","Directrouting": true}}' [root@hdss7-21 etcd]# supervisor start flanneld-7-21 [root@hdss7-22 ~]# supervisor start flanneld-7-22
在各运算节点上优化iptables规则
- 优化SNAT规则，各运算节点之间的各POD之间的网络通信不再出网
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE [root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
10.4.7.21主机上的，来源是172.7.21.0/24段的docker的ip，目标ip不是172.7.0.0/16段，网络发包不从docker0桥设备出站的，才进行SNAT转换
各运算节点保存iptables规则
[root@hdss7-21 ~]# service iptables save
、
