docker flannel网络部署和路由走向分析
1.flannel介绍
flannel是coreos开发的容器网络解决方案。flannel为每个host分配一个subnet，容器从此subnet中分配ip。这些ip可以在host间路由，容器间无需nat和port mapping就可以跨主机通讯。
每个subnet都是从一个更大的ip池中划分的，flannel会在每个主机上运行一个叫flanneld得agent，其职责是从ip池中分配subnet。为了在各个主机间共享信息，flannel用etcd存放网络配置，已分配的subnet，host的ip等信息。
数据包通过backend在主机间转发。
flannel提供了多种backend，最常用的有vxlan和host-gw。
2.部署实验环境
三个虚机
docker1 docker2 docker3?
etcd安装在docker1
docker1 docker2 docker3上运行flanneld
注:为了更方便的验证flannel和etcd所以docker1也安装了flannel，
其实可以不用在docker1安装
centos7自带了软件包，直接yum安装即可
2.1?
安装配置etcd
yum -y install etcd
[root@docker1 ~]# systemctl start etcd? && systemctl enable etcd
[root@docker1 ~]#测试下
[root@docker1 ~]# etcd --version
etcd Version: 3.2.18
Git SHA: eddf599
Go Version: go1.9.4
Go OS/Arch: linux/amd64
[root@docker1 ~]#
[root@docker1 ~]# etcdctl set test "a"
a
[root@docker1 ~]# etcdctl get test
a
[root@docker1 ~]#2.2?
安装配置flannel
[root@docker1 ~]# yum -y install flannel启动
[root@docker1 ~]# systemctl start flanneld报错
[root@docker1 ~]# systemctl status flanneld -l
● flanneld.service - Flanneld overlay address etcd agent
?? Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
?? Active: activating (start) since Thu 2018-06-14 02:22:26 EDT; 1min 1s ago
Main PID: 2950 (flanneld)
?? Memory: 16.6M
?? CGroup: /system.slice/flanneld.service
? ? ? ? ?? └─2950 /usr/bin/flanneld -etcd-endpoints=http://127.0.0.1:2379 -etcd-prefix=/atomic.io/network
Jun 14 02:23:18 docker1 flanneld-start[2950]: E0614 02:23:18.974351? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:19 docker1 flanneld-start[2950]: E0614 02:23:19.977497? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:20 docker1 flanneld-start[2950]: E0614 02:23:20.980721? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:21 docker1 flanneld-start[2950]: E0614 02:23:21.983553? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:22 docker1 flanneld-start[2950]: E0614 02:23:22.988446? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:23 docker1 flanneld-start[2950]: E0614 02:23:23.992106? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:24 docker1 flanneld-start[2950]: E0614 02:23:24.994719? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:25 docker1 flanneld-start[2950]: E0614 02:23:25.998629? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:27 docker1 flanneld-start[2950]: E0614 02:23:27.002486? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]
Jun 14 02:23:28 docker1 flanneld-start[2950]: E0614 02:23:28.006185? ? 2950 network.go:102] failed to retrieve network config: 100: Key not found (/atomic.io) [11]注意-etcd-prefix=/automic.io/network
flanel读取的网络配置是这个文件，这个文件是在
[root@docker1 ~]# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service[root@docker1 sysconfig]# cat flanneld
# Flanneld configuration options?
# etcd url location.? Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://127.0.0.1:2379"
# etcd config key.? This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/atomic.io/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""注意：
FLANNEL_ETCD_PREFIX="/atomic.io/network"
这个FLANNEL_ETCD_PREFIX需要etcdctl手动去建立
[root@docker1 ~]# etcdctl mk /atomic.io/network/config ‘{"Network":"172.17.0.0/16", "SubnetMin": "172.17.1.0", "SubnetMax": "172.17.254.0", "Backend":{"Type":"vxlan"}}‘再启动flannel，启动正常
[root@docker1 ~]# systemctl start flanneld && systemctl enable flanneld
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
Created symlink from /etc/systemd/system/docker.service.requires/flanneld.service to /usr/lib/systemd/system/flanneld.service.
[root@docker1 ~]#[root@docker1 ~]# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
?? Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
?? Active: active (running) since Thu 2018-06-14 02:47:58 EDT; 11s ago
? Process: 3513 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
Main PID: 3475 (flanneld)
?? Memory: 18.5M
?? CGroup: /system.slice/flanneld.service
? ? ? ? ?? └─3475 /usr/bin/flanneld -etcd-endpoints=http://127.0.0.1:2379 -etcd-prefix=/atomic.io/network
Jun 14 02:47:52 docker1 flanneld-start[3475]: E0614 02:47:52.150129? ? 3475 network.go:102] failed to retrieve network co...) [14]
Jun 14 02:47:53 docker1 flanneld-start[3475]: E0614 02:47:53.152602? ? 3475 network.go:102] failed to retrieve network co...) [14]
Jun 14 02:47:54 docker1 flanneld-start[3475]: E0614 02:47:54.155402? ? 3475 network.go:102] failed to retrieve network co...) [14]
Jun 14 02:47:55 docker1 flanneld-start[3475]: E0614 02:47:55.158612? ? 3475 network.go:102] failed to retrieve network co...) [14]
Jun 14 02:47:56 docker1 flanneld-start[3475]: E0614 02:47:56.164481? ? 3475 network.go:102] failed to retrieve network co...) [14]
Jun 14 02:47:57 docker1 flanneld-start[3475]: E0614 02:47:57.168282? ? 3475 network.go:102] failed to retrieve network co...) [14]
Jun 14 02:47:58 docker1 flanneld-start[3475]: I0614 02:47:58.179298? ? 3475 local_manager.go:179] Picking subnet in range....254.0
Jun 14 02:47:58 docker1 flanneld-start[3475]: I0614 02:47:58.261220? ? 3475 manager.go:250] Lease acquired: 172.17.21.0/24
Jun 14 02:47:58 docker1 flanneld-start[3475]: I0614 02:47:58.261993? ? 3475 network.go:98] Watching for new subnet leases
Jun 14 02:47:58 docker1 systemd[1]: Started Flanneld overlay address etcd agent.
Hint: Some lines were ellipsized, use -l to show in full.看看这个脚本
Process: 3513 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
flannel_env="/run/flannel/subnet.env"
docker_env="/run/docker_opts.env"
combined_opts_key="DOCKER_OPTS"
indiv_opts=false
combined_opts=false
ipmasq=true检查下文件内容，我感觉是根据这个文件来生成网段，不确认
[root@docker1 flannel]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.17.0.0/16
FLANNEL_SUBNET=172.17.21.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=false看看ip段
[root@docker1 ~]# ip a |grep flannel
11: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
? ? inet 172.17.21.0/16 scope global flannel0
[root@docker1 ~]#以上都是docker1上的操作
2.3?
docker2,docker3上的操作是一样的，我记录docker2上的操作
[root@docker2 ~]# yum -y install flannel启动flannel
[root@docker2 ~]# flanneld -etcd-endpoints=http://192.168.211.140:2379 -iface=ens33 -etcd-prefix=/atomic.io/network
I0614 04:28:55.785204? ? 2767 main.go:132] Installing signal handlers
I0614 04:28:55.785764? ? 2767 manager.go:149] Using interface with name ens33 and address 192.168.211.154
I0614 04:28:55.785784? ? 2767 manager.go:166] Defaulting external address to interface address (192.168.211.154)
E0614 04:28:55.786742? ? 2767 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.168.211.140:2379: getsockopt: no route to host
E0614 04:28:57.788671? ? 2767 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.168.211.140:2379: i/o timeout
E0614 04:28:59.791359? ? 2767 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.168.211.140:2379: i/o timeout报错了
这个错误是因为etcd默认只监听本机的2379端口
[root@docker1 ~]# cat /etc/etcd/etcd.conf
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="default"
#ETCD_SNAPSHOT_COUNT="100000"把ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"改成ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
重新启动etcd
[root@docker1 ~]# systemctl restart etcd
[root@docker1 ~]# systemctl status etcd
● etcd.service - Etcd Server
?? Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
?? Active: active (running) since Thu 2018-06-14 04:38:49 EDT; 1min 11s ago
Main PID: 3401 (etcd)
?? Memory: 21.5M
?? CGroup: /system.slice/etcd.service
? ? ? ? ?? └─3401 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://0.0.0.0:23...
Jun 14 04:38:48 docker1 etcd[3401]: enabled capabilities for version 3.2
Jun 14 04:38:49 docker1 etcd[3401]: 8e9e05c52164694d is starting a new election at term 9
Jun 14 04:38:49 docker1 etcd[3401]: 8e9e05c52164694d became candidate at term 10
Jun 14 04:38:49 docker1 etcd[3401]: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 10
Jun 14 04:38:49 docker1 etcd[3401]: 8e9e05c52164694d became leader at term 10
Jun 14 04:38:49 docker1 etcd[3401]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 10
Jun 14 04:38:49 docker1 etcd[3401]: published {Name:default ClientURLs:[http://192.168.211.140:2379]} to cluster cdf8...3a8c32
Jun 14 04:38:49 docker1 etcd[3401]: ready to serve client requests
Jun 14 04:38:49 docker1 systemd[1]: Started Etcd Server.
Jun 14 04:38:49 docker1 etcd[3401]: serving insecure client requests on [::]:2379, this is strongly discouraged!
Hint: Some lines were ellipsized, use -l to show in full.
[root@docker1 ~]#再启动还是报错?
[root@docker2 ~]# systemctl status flanneld -l
● flanneld.service - Flanneld overlay address etcd agent
?? Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
?? Active: inactive (dead)
Jun 14 04:21:53 docker2 flanneld-start[2706]: E0614 04:21:53.879476? ? 2706 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 14 04:21:54 docker2 flanneld-start[2706]: E0614 04:21:54.880962? ? 2706 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 14 04:21:55 docker2 flanneld-start[2706]: E0614 04:21:55.882332? ? 2706 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 14 04:21:56 docker2 flanneld-start[2706]: E0614 04:21:56.887002? ? 2706 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 14 04:21:57 docker2 flanneld-start[2706]: E0614 04:21:57.888246? ? 2706 network.go:102] failed to retrieve network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: getsockopt: connection refused
Jun 14 04:21:58 docker2 flanneld-start[2706]: E0614 04:21:58.889903? ? 2706 network.go:102] failed to retrieve network config: client: etcd cluster is