k8s flannel docker0 IP问题记录

故障现象：

当flannel服务重启后，会分配新的ip地址，分别更新etcd库和subnet.env文件

/etcd/bin/etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.66.0-24
/coreos.com/network/subnets/10.0.38.0-24

[root@k8s-221 ~]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=10.0.38.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.0.38.1/24 --ip-masq=false --mtu=1450"

路由也没有正常更新，导致服务网络异常

[root@k8s-221 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 ens33
10.0.20.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
10.0.66.0 10.0.66.0 255.255.255.0 UG 0 0 0 flannel.1
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.1.230 0.0.0.0 255.255.255.255 UH 0 0 0 lo

此时即使重启docker服务后，查看docker0网卡地址仍然是原来的IP。

[root@k8s-221 ~]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 10.0.20.1 netmask 255.255.255.0 broadcast 10.0.20.255
ether 02:42:21:3f:c7:f8 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

此时最好的方法就是重启系统，网上的删除网卡brctl delbr docker0方法并不好用。尽量保证不重启flannel服务。

系统重启后IP正常

[root@k8s-221 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.0.38.1 netmask 255.255.255.0 broadcast 10.0.38.255
inet6 fe80::42:67ff:fe52:597 prefixlen 64 scopeid 0x20<link>
ether 02:42:67:52:05:97 txqueuelen 0 (Ethernet)
RX packets 13560 bytes 2363957 (2.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 13860 bytes 13485610 (12.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.221 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::20c:29ff:fe7c:be8e prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7c:be:8e txqueuelen 1000 (Ethernet)
RX packets 490363 bytes 527927171 (503.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 635362 bytes 733366309 (699.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.0.38.0 netmask 255.255.255.255 broadcast 10.0.38.0
inet6 fe80::a043:f1ff:feda:930f prefixlen 64 scopeid 0x20<link>
ether a2:43:f1:da:93:0f txqueuelen 0 (Ethernet)
RX packets 1829 bytes 1091538 (1.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1567 bytes 1093218 (1.0 MiB)
TX errors 0 dropped 5 overruns 0 carrier 0 collisions 0

路由也恢复正常
[root@k8s-221 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 ens33
10.0.38.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
10.0.66.0 10.0.66.0 255.255.255.0 UG 0 0 0 flannel.1
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33