修复tunl0-二进制安装calico
这篇博文很重要，出现这个问题导致pod之间无法通讯，pod无法连接外网。
出现的问题是二进制方式安装了节点之后， tunl0没有显示，通过ifconfig tunl0 up 启动tunl0 没有意义，
只有通过 calico 启动ipip模式，自动分配ip 才能解决这个问题 , 如果tunl0,没有分配到ip,也就意味着他不能跟其他pod通讯，在该node上创建的pod内部不能访问外网。
所以重点就是要解决tunl0,为什么没有启用和分配到集群的ip问题。是本文的重点
本机安装环境
k8s节点是通过二进制方式安装。
[root@ht22 calico]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@ht22 calico]# uname -r
3.10.0-1160.45.1.el7.x86_64
[root@ht22 calico]# docker -v
Docker version 18.03.0-ce, build 0520e24
[root@ht22 calico]# docker -v
Docker version 18.03.0-ce, build 0520e24
[root@ht22 calico]# ifconfig
cali1e4a9cee8dc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 568108 bytes 328205511 (313.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 568108 bytes 328205511 (313.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:83:9e:06:86 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.51.204 netmask 255.255.255.0 broadcast 10.129.51.255
ether 06:34:a0:00:0b:81 txqueuelen 1000 (Ethernet)
RX packets 401212 bytes 136527652 (130.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 207123 bytes 197287098 (188.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.53.204 netmask 255.255.255.0 broadcast 10.129.53.255
ether 06:0f:2e:00:0d:03 txqueuelen 1000 (Ethernet)
RX packets 35636 bytes 2139100 (2.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 1421 (1.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 568108 bytes 328205511 (313.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 568108 bytes 328205511 (313.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tunl0: flags=193<UP,RUNNING,NOARP> mtu 1440
tunnel txqueuelen 1000 (IPIP Tunnel)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
解决方式如下：
[root@ht22 calico]# docker rm -f calico-node
[root@ht5 calico]# docker ps | grep calico-node
1effd55ba758 quay.io/calico/node:v3.1.0 "start_runit" 5 minutes ago Up 5 minutes calico-node
d8aefa455a37 9f355e076ea7 "/install-cni.sh" 2 months ago Up 2 months k8s_install-cni_calico-node-ld4pd_kube-system_249f9a81-1c3d-11ec-9a8a-065c62000e9f_1
691f562030f4 k8s.gcr.io/pause-amd64:3.0 "/pause" 2 months ago Up 2 months k8s_POD_calico-node-ld4pd_kube-system_249f9a81-1c3d-11ec-9a8a-065c62000e9f_1
进入到calico安装目录
[root@ht22 calico]# ll
total 392880
drwxr-xr-x 2 root root 4096 Nov 26 20:38 calico
-rw-r--r-- 1 root root 11829 Sep 26 2019 calico.example.yaml
-rw-r--r-- 1 root root 2483 Feb 17 11:33 calico-node.service.sh
-rw-r--r-- 1 root root 277872640 Nov 26 20:38 caliconode.v3.1.0.tar
-rw-r--r-- 1 root root 1081 Nov 26 20:38 calico.sh
-rw-r--r-- 1 root root 19389 Sep 26 2019 calico.yaml
-rw-r--r-- 1 root root 69124608 Nov 26 20:38 cni.v3.1.3.tar
-rw-r--r-- 1 root root 55256576 Nov 26 20:38 kube-controllers.v3.1.3.tar
-rw-r--r-- 1 root root 1247 Nov 26 20:38 rbac.yaml
//删除前
[root@ht5 calico]# docker ps | grep calico-node
1effd55ba758 quay.io/calico/node:v3.1.0 "start_runit" 5 minutes ago Up 5 minutes calico-node
d8aefa455a37 9f355e076ea7 "/install-cni.sh" 2 months ago Up 2 months k8s_install-cni_calico-node-ld4pd_kube-system_249f9a81-1c3d-11ec-9a8a-065c62000e9f_1
691f562030f4 k8s.gcr.io/pause-amd64:3.0 "/pause" 2 months ago Up 2 months k8s_POD_calico-node-ld4pd_kube-system_249f9a81-1c3d-11ec-9a8a-065c62000e9f_1
//删除后
[root@ht22 calico]# docker ps | grep calico-node
2f6942eaad74 9f355e076ea7 "/install-cni.sh" 9 hours ago Up 9 hours k8s_install-cni_calico-node-r92zw_kube-system_5e332799-8f14-11ec-b44a-060eb4000e9d_1
27dfc117821e k8s.gcr.io/pause-amd64:3.0 "/pause" 9 hours ago Up 9 hours k8s_POD_calico-node-r92zw_kube-system_5e332799-8f14-11ec-b44a-060eb4000e9d_1
//重新安装，整个过程就是
[root@ht22 calico]# docker rm -f calico-node
//进入到k8snode安装包，进入calico目录
[root@ht22 calico]# sh calico-node.service.sh
[root@ht22 calico]# systemctl daemon-reload
[root@ht22 calico]# systemctl start calico-node
[root@ht22 calico]# ifconfig
cali1e4a9cee8dc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 569135 bytes 328540747 (313.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 569135 bytes 328540747 (313.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:83:9e:06:86 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.51.204 netmask 255.255.255.0 broadcast 10.129.51.255
ether 06:34:a0:00:0b:81 txqueuelen 1000 (Ethernet)
RX packets 408912 bytes 139184171 (132.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 211608 bytes 202926820 (193.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.53.204 netmask 255.255.255.0 broadcast 10.129.53.255
ether 06:0f:2e:00:0d:03 txqueuelen 1000 (Ethernet)
RX packets 35741 bytes 2145400 (2.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 1421 (1.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 569135 bytes 328540747 (313.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 569135 bytes 328540747 (313.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tunl0: flags=193<UP,RUNNING,NOARP> mtu 1440
inet 172.17.31.0 netmask 255.255.255.255
tunnel txqueuelen 1000 (IPIP Tunnel)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
附 [root@ht22 calico]# cat calico-node.service.sh
K8S_MASTER_IP="这里是master ip"
HOSTNAME='本机hostname'
#HOSTNAME=`cat /etc/hostname`
ETCD_ENDPOINTS="https://10.121.5.11:2379,https://10.121.5.12:2379"
cat > /lib/systemd/system/calico-node.service <<EOF
[Unit]
Description=calico node
After=docker.service
Requires=docker.service
[Service]
User=root
Environment=ETCD_ENDPOINTS=${ETCD_ENDPOINTS}
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \\
-e ETCD_ENDPOINTS=\${ETCD_ENDPOINTS} \\
-e ETCD_CA_CERT_FILE=/etc/etcd/ssl/etcd-root-ca.pem \\
-e ETCD_CERT_FILE=/etc/etcd/ssl/etcd.pem \\
-e ETCD_KEY_FILE=/etc/etcd/ssl/etcd-key.pem \\
-e NODENAME=${HOSTNAME} \\
-e IP= \\
-e IP_AUTODETECTION_METHOD=can-reach=${K8S_MASTER_IP} \\
-e AS=64512 \\
-e CLUSTER_TYPE=k8s,bgp \\
-e CALICO_IPV4POOL_CIDR=172.17.0.0/16 \\
-e CALICO_IPV4POOL_IPIP=always \\
-e CALICO_LIBNETWORK_ENABLED=true \\
-e CALICO_NETWORKING_BACKEND=bird \\
-e CALICO_DISABLE_FILE_LOGGING=true \\
-e FELIX_IPV6SUPPORT=false \\
-e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \\
-e FELIX_LOGSEVERITYSCREEN=info \\
-e FELIX_IPINIPMTU=1440 \\
-e FELIX_HEALTHENABLED=true \\
-e CALICO_K8S_NODE_REF=${HOSTNAME} \\
-v /etc/calico/etcd-root-ca.pem:/etc/etcd/ssl/etcd-root-ca.pem \\
-v /etc/calico/etcd.pem:/etc/etcd/ssl/etcd.pem \\
-v /etc/calico/etcd-key.pem:/etc/etcd/ssl/etcd-key.pem \\
-v /lib/modules:/lib/modules \\
-v /var/lib/calico:/var/lib/calico \\
-v /var/run/calico:/var/run/calico \\
quay.io/calico/node:v3.1.0
ExecStop=/usr/bin/docker rm -f calico-node
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
//我们在查看下
[root@ht22 calico]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.129.51.1 0.0.0.0 UG 0 0 0 ens192
10.121.51.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192
10.121.53.0 0.0.0.0 255.255.255.0 U 100 0 0 ens224
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 ens192
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.17.9.128 10.128.51.14 255.255.255.192 UG 0 0 0 tunl0
172.17.23.128 10.129.52.191 255.255.255.192 UG 0 0 0 tunl0
172.17.23.192 10.129.55.113 255.255.255.192 UG 0 0 0 tunl0
172.17.27.64 10.129.55.112 255.255.255.192 UG 0 0 0 tunl0
172.17.31.0 0.0.0.0 255.255.255.192 U 0 0 0 *
172.17.50.192 10.129.55.111 255.255.255.192 UG 0 0 0 tunl0
172.17.157.0 10.128.51.12 255.255.255.192 UG 0 0 0 tunl0
172.17.168.0 10.129.52.193 255.255.255.192 UG 0 0 0 tunl0
172.17.216.64 10.129.52.192 255.255.255.192 UG 0 0 0 tunl0
172.17.246.64 10.128.51.11 255.255.255.192 UG 0 0 0 tunl0
//tunl0 出现，否则这里也不会出现tunl0信息。
注意： 最后一步就是所有在其上的pod都要重新创建，否则还是网络不了外网。
