cali1e4a9cee8dc这是什么东西？

//我们查下k8s node节点，发现很多类似  cali7c620a7a67b 这样的类似网络设备的东西。
//这些是什么呢？

//k8s集群节点ht10，node网络情况。
[root@ht10 calico]# ifconfig

cali1e4a9cee8dc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 495759 bytes 283173556 (270.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 495759 bytes 283173556 (270.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
//这个就是对应 [root@ht10 calico]#  ip  a 命令结果里面的网卡编号77

cali23eb546861c: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

cali72e0f3b6f9b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 33001 bytes 1980958 (1.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 1421 (1.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

caliaa64c1a7f09: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

calica7f2079b57: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:83:9e:06:86 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.51.204 netmask 255.255.255.0 broadcast 10.129.51.255
ether 06:34:a0:00:0b:81 txqueuelen 1000 (Ethernet)
RX packets 215955 bytes 65222322 (62.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 110081 bytes 77325192 (73.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.129.53.204 netmask 255.255.255.0 broadcast 10.129.53.255
ether 06:0f:2e:00:0d:03 txqueuelen 1000 (Ethernet)
RX packets 33001 bytes 1980958 (1.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 1421 (1.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 495759 bytes 283173556 (270.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 495759 bytes 283173556 (270.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

 

没有安装成k8s节点的普通机器上采样应该就是,去掉calico,docker，那么ifconfig结果如下

[root@普通机器 ~]# ifconfig
eth1 Link encap:Ethernet HWaddr 00:50:56:A2:78:01 
inet addr:10.129.55.19 Bcast:10.129.55.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fea2:7801/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2197601980 errors:0 dropped:0 overruns:0 frame:0
TX packets:1940531676 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:848519208193 (790.2 GiB) TX bytes:815714904879 (759.6 GiB)

lo Link encap:Local Loopback 
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:46657 errors:0 dropped:0 overruns:0 frame:0
TX packets:46657 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 
RX bytes:2622944 (2.5 MiB) TX bytes:2622944 (2.5 MiB)

tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 
inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:52063234 errors:0 dropped:0 overruns:0 frame:0
TX packets:56440262 errors:0 dropped:747 overruns:0 carrier:0
collisions:0 txqueuelen:100 
RX bytes:3556025519 (3.3 GiB) TX bytes:14467954568 (13.4 GiB)  

//这个tun0可有可无。

 

我们分析如下：

1、centos7的网卡命名已经变成了 ens* 类似的， 不再是centos6的 eth* 这样的方式
   我们的服务器基本都是eth0,eth1,ens* 这种形式的。 当然有的机器可能有网卡信息，但是对应配置文件没有情况也非常常见。
2、ens192和ens224是两个网卡，即该机器有两个实际网卡
3、tunl0，百度一下可以看到是ipip隧道, 就是calico利用tunl0来进行通讯，同时安装calico的时候分配ip.

[root@ht22 k8snode]# modinfo ipip
filename: /lib/modules/3.10.0-1160.45.1.el7.x86_64/kernel/net/ipv4/ipip.ko.xz
alias: netdev-tunl0
alias: rtnl-link-ipip
license: GPL
retpoline: Y
rhelversion: 7.9
srcversion: 8032CC3EDB2F63D42025A07
depends: ip_tunnel,tunnel4
intree: Y
vermagic: 3.10.0-1160.45.1.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: F5:EF:E0:5F:FF:E7:34:59:FB:55:F6:1B:3A:BD:2F:18:1F:8E:51:A0
sig_hashalgo: sha256
parm: log_ecn_error:Log packets received with corrupted ECN (bool)

 4、lo 本地回环接口（网络接口，注意区别loop 叫回环设备），这个是针对网络通讯而言
 5、docker0,在docker安装之后,就会产生一个docker0的虚拟网桥

 ip a 方式查看网卡编号

[root@ht10 calico]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 06:34:a0:00:0b:81 brd ff:ff:ff:ff:ff:ff
inet 10.129.51.204/24 brd 10.129.51.255 scope global ens192
valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 06:0f:2e:00:0d:03 brd ff:ff:ff:ff:ff:ff
inet 10.129.53.204/24 brd 10.129.53.255 scope global noprefixroute ens224
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:83:9e:06:86 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
71: cali23eb546861c@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3
75: tunl0@NONE: <NOARP> mtu 1480 qdisc noqueue state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
76: calica7f2079b57@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
77: cali1e4a9cee8dc@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
78: caliaa64c1a7f09@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
79: cali72e0f3b6f9b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 4

注意： link-netnsid 为跨网络接口设置对等网络标识

单独看下这个设备，对于这个设备来说，这其实是一个veth peer：

[root@ht5 calico]# ip -d link show dev cali7c620a7a67b
119: cali7c620a7a67b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3 promiscuity 0
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

我们看到最后一行有veth字样，代表这是一个veth设备，对这个@if4来说，这表示 link’s peer 端对端端口的索引。尽管这个属性似乎对任何接口都可用，但它只适用于几种接口类型:veth、macvlan、vlan(子接口)，表示和另一个接口的关系。

 

我们下面做个实验，看下pod对应的宿主机的网卡是什么，宿主机上创建容器(pod里的)对应网卡是什么？

pod是通过master或dashborad等创建的。

1、master上执行操作

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

[root@master-38 ~]# kubectl get pods -n cc -o wide NAME READY STATUS RESTARTS AGE IP NODE  mdp-cc-85fd974f8c-j2q8v 1/1 Running 0 3h 172.17.76.28 ht10    //没有加命名空间，所以出错 [root@master-38 ~]# kubectl exec mdp-cc-85fd974f8c-j2q8v -it -- /bin/bash Error from server (NotFound): pods "   p-cc-85fd974f8c-j2q8v" not found //加上命名空间，进入pod [root@master-38 ~]# kubectl exec mdp-cc-85fd974f8c-j2q8v -it -n cc -- /bin/bash root@mdp-cc-85fd974f8c-j2q8v:/# ip a   //进入pod里面 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 0e:62:0d:13:27:03 brd ff:ff:ff:ff:ff:ff inet 172.17.76.28/32 scope global eth0 valid_lft forever preferred_lft forever //注意77就是宿主机网卡设备的编号

2、现在我们登录宿主机ht10这台机器。

[root@ht10 net.d]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 06:34:a0:00:0b:81 brd ff:ff:ff:ff:ff:ff
inet 10.129.51.204/24 brd 10.129.51.255 scope global ens192
valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 06:0f:2e:00:0d:03 brd ff:ff:ff:ff:ff:ff
inet 10.129.53.204/24 brd 10.129.53.255 scope global noprefixroute ens224
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
link/ether 02:42:83:9e:06:86 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
71: cali23eb546861c@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3
75: tunl0@NONE: <NOARP> mtu 1480 qdisc noqueue state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
76: calica7f2079b57@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
77: cali1e4a9cee8dc@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1  
 //pod这个就是对应的网络设备   link-netnsid 1：所在网络命名空间的id为1，也就是对端在netnsid为1的网络命名空间里
78: caliaa64c1a7f09@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
79: cali72e0f3b6f9b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 4

//执行ip a 或者ip addr 命令之后，经过前面有编号的，就不是很乱了，cali7c620a7a67b

  //link-netnsid 为跨网络接口设置对等网络标识

 

3、docker容器和网卡的对应关系:

[root@ht10 net.d]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
22da71747c54 registry.aaa.com.cn/cc/test/mdp-cc "java -jar /mdp-cc…" 18 minutes ago Up 18 minutes

容器 id 是 22da71747c54，进入容器

[root@ht10 net.d]# docker exec -it 22da71747c54 /bin/bash

root@mdp-cc-85fd974f8c-j2q8v:/# cat /sys/class/net/eth0/iflink  //容器内输入命令查看
77
这里我们看到是77。