failed to allocate for range 0: no IP addresses available in range set: 172.20.xx.1-172.20.xx.254

今天遇到一个机器上的Pod 在创建以后一直处于Init 0/1的状态，进到这个节点查看其kubelet的状态，发现果然有问题

systemctl status kubelet

 

.go:77] Container "1a7183767162740ba734e2c4b880e2937af1680e15fb1a7d66dc7e48fe0ca68c" not found in pod's containers
 RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "ts-train-mongo-7f6f6668bd-rn2n4_default" network: failed to allocate f
o:54] CreatePodSandbox for pod "ts-train-mongo-7f6f6668bd-rn2n4_default(1264e4b7-fd27-11e8-aa60-525400c4f9f5)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up 
o:647] createPodSandbox for pod "ts-train-mongo-7f6f6668bd-rn2n4_default(1264e4b7-fd27-11e8-aa60-525400c4f9f5)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up
rror syncing pod 1264e4b7-fd27-11e8-aa60-525400c4f9f5 ("ts-train-mongo-7f6f6668bd-rn2n4_default(1264e4b7-fd27-11e8-aa60-525400c4f9f5)"), skipping: failed to "CreatePodSandbox" for "ts-tra
o:403] No ready sandbox for pod "ts-order-other-service-78d5ff8b57-2k6gf_default(830f4782-fd27-11e8-aa60-525400c4f9f5)" can be found. Need to start a new one
o:403] No ready sandbox for pod "ts-station-service-c9ff7c7b7-pkpt9_default(9a9c5010-fd27-11e8-aa60-525400c4f9f5)" can be found. Need to start a new one
o:403] No ready sandbox for pod "ts-preserve-service-7b95474f77-8l4jw_default(88bc030d-fd27-11e8-aa60-525400c4f9f5)" can be found. Need to start a new one
ing network: failed to allocate for range 0: no IP addresses available in range set: 172.20.5.1-172.20.5.254
le adding to cni network: failed to allocate for range 0: no IP addresses available in range set: 172.20.5.1-172.20.5.254

 

 提示200多个IP都被用完了，这什么情况

/var/lib/cni/networks/cbr0# ls
172.20.5.10   172.20.5.118  172.20.5.136  172.20.5.154  172.20.5.172  172.20.5.190  172.20.5.208  172.20.5.226  172.20.5.244  172.20.5.33  172.20.5.51  172.20.5.7   172.20.5.88
172.20.5.100  172.20.5.119  172.20.5.137  172.20.5.155  172.20.5.173  172.20.5.191  172.20.5.209  172.20.5.227  172.20.5.245  172.20.5.34  172.20.5.52  172.20.5.70  172.20.5.89
172.20.5.101  172.20.5.12   172.20.5.138  172.20.5.156  172.20.5.174  172.20.5.192  172.20.5.21   172.20.5.228  172.20.5.246  172.20.5.35  172.20.5.53  172.20.5.71  172.20.5.9
172.20.5.102  172.20.5.120  172.20.5.139  172.20.5.157  172.20.5.175  172.20.5.193  172.20.5.210  172.20.5.229  172.20.5.247  172.20.5.36  172.20.5.54  172.20.5.72  172.20.5.90
172.20.5.103  172.20.5.121  172.20.5.14   172.20.5.158  172.20.5.176  172.20.5.194  172.20.5.211  172.20.5.23   172.20.5.248  172.20.5.37  172.20.5.55  172.20.5.73  172.20.5.91
172.20.5.104  172.20.5.122  172.20.5.140  172.20.5.159  172.20.5.177  172.20.5.195  172.20.5.212  172.20.5.230  172.20.5.249  172.20.5.38  172.20.5.56  172.20.5.74  172.20.5.92
172.20.5.105  172.20.5.123  172.20.5.141  172.20.5.16   172.20.5.178  172.20.5.196  172.20.5.213  172.20.5.231  172.20.5.25   172.20.5.39  172.20.5.57  172.20.5.75  172.20.5.93
172.20.5.106  172.20.5.124  172.20.5.142  172.20.5.160  172.20.5.179  172.20.5.197  172.20.5.214  172.20.5.232  172.20.5.250  172.20.5.4   172.20.5.58  172.20.5.76  172.20.5.94
172.20.5.107  172.20.5.125  172.20.5.143  172.20.5.161  172.20.5.18   172.20.5.198  172.20.5.215  172.20.5.233  172.20.5.251  172.20.5.40  172.20.5.59  172.20.5.77  172.20.5.95
172.20.5.108  172.20.5.126  172.20.5.144  172.20.5.162  172.20.5.180  172.20.5.199  172.20.5.216  172.20.5.234  172.20.5.252  172.20.5.41  172.20.5.6   172.20.5.78  172.20.5.96
172.20.5.109  172.20.5.127  172.20.5.145  172.20.5.163  172.20.5.181  172.20.5.2    172.20.5.217  172.20.5.235  172.20.5.253  172.20.5.42  172.20.5.60  172.20.5.79  172.20.5.97
172.20.5.11   172.20.5.128  172.20.5.146  172.20.5.164  172.20.5.182  172.20.5.20   172.20.5.218  172.20.5.236  172.20.5.254  172.20.5.43  172.20.5.61  172.20.5.8   172.20.5.98
172.20.5.110  172.20.5.129  172.20.5.147  172.20.5.165  172.20.5.183  172.20.5.200  172.20.5.219  172.20.5.237  172.20.5.26   172.20.5.44  172.20.5.62  172.20.5.80  172.20.5.99
172.20.5.111  172.20.5.13   172.20.5.148  172.20.5.166  172.20.5.184  172.20.5.201  172.20.5.22   172.20.5.238  172.20.5.27   172.20.5.45  172.20.5.63  172.20.5.81  172.20.6.197
172.20.5.112  172.20.5.130  172.20.5.149  172.20.5.167  172.20.5.185  172.20.5.202  172.20.5.220  172.20.5.239  172.20.5.28   172.20.5.46  172.20.5.64  172.20.5.82  last_reserved_ip.0
172.20.5.113  172.20.5.131  172.20.5.15   172.20.5.168  172.20.5.186  172.20.5.203  172.20.5.221  172.20.5.24   172.20.5.29   172.20.5.47  172.20.5.65  172.20.5.83  lock
172.20.5.114  172.20.5.132  172.20.5.150  172.20.5.169  172.20.5.187  172.20.5.204  172.20.5.222  172.20.5.240  172.20.5.3    172.20.5.48  172.20.5.66  172.20.5.84
172.20.5.115  172.20.5.133  172.20.5.151  172.20.5.17   172.20.5.188  172.20.5.205  172.20.5.223  172.20.5.241  172.20.5.30   172.20.5.49  172.20.5.67  172.20.5.85
172.20.5.116  172.20.5.134  172.20.5.152  172.20.5.170  172.20.5.189  172.20.5.206  172.20.5.224  172.20.5.242  172.20.5.31   172.20.5.5   172.20.5.68  172.20.5.86
172.20.5.117  172.20.5.135  172.20.5.153  172.20.5.171  172.20.5.19   172.20.5.207  172.20.5.225  172.20.5.243  172.20.5.32   172.20.5.50  172.20.5.69  172.20.5.87

##flannel创建了很多文件
/var/lib/cni/flannel#  ls | wc ; date 

 

解决方案

rm -rf /var/lib/cni/flannel/* && rm -rf /var/lib/cni/networks/cbr0/* && ip link delete cni0
rm -rf  /var/lib/cni/networks/cni0/*