056、macvlan网络结构分析（2019-03-25 周一）
macvlan不依赖linux bridge
brctl show 可以确认没有创建新的bridge
查看容器中只有一块网卡 eth0@if3 ，对应host上的 3号接口
容器的interface 直接与host的网卡连接，这种方法使得容器无需通过NAT和端口映射就能与外网直接通信（只要网络中有网关），在网络上与其他独立的主机没有区别
root@host1:~# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242a29df713 no
root@host1:~# docker exec bbox1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: eth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:10:56:0b brd ff:ff:ff:ff:ff:ff
root@host1:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:4c:70 brd ff:ff:ff:ff:ff:ff
3: ens192: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:22:32 brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:a2:9d:f7:13 brd ff:ff:ff:ff:ff:ff
用 sub-interface实现多macvlan网络
macvlan会独占主机的网卡，也就是说一个网卡只能创建一个macvlan网络，否则会报错
root@host1:~# docker network create -d macvlan --subnet 172.16.87.0/24 --gateway 172.16.87.1 -o parent=ens192 mac_net2
Error response from daemon: network dm-d60df792c936 is already using parent interface ens192
但是主机的网卡数量是有限的，如何支持更多的macvlan网络呢？
好在macvlan不仅可以连接到 interface （ens192），还可以连接到 sub-interface （ens192.xxx）
VLAN是现代网络常用的网络虚拟化技术，他可以将物理的二层网络划分成多达4094个逻辑网络，这些逻辑网络在二层上是相互隔离的，每个逻辑网络（即VLAN）由 VLAN ID 区分，VLAN ID 的取值 1 - 4094
Linux的网卡也能支持VLAN（apt-get install vlan），同一个interface可以收发多个VLAN的数据包，不过前提是要创建VLAN的sub-interface
比如希望ens192 同时支持vlan10 和vlan20，则需创建sub-interface ens192.10 和 ens192.20
在交换机上，如果某个port只能收发单个VLAN的数据，该port为Access模式。如果支持多VLAN，则为Trunk模式
root@host1:~# apt-get install vlan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
vlan
1 upgraded, 0 newly installed, 0 to remove and 125 not upgraded.
Need to get 30.7 kB of archives.
After this operation, 45.1 kB disk space will be freed.
Get:1 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 vlan amd64 1.9-3.2ubuntu1.16.04.5 [30.7 kB]
Fetched 30.7 kB in 5s (5,469 B/s)
(Reading database ... 60147 files and directories currently installed.)
Preparing to unpack .../vlan_1.9-3.2ubuntu1.16.04.5_amd64.deb ...
Unpacking vlan (1.9-3.2ubuntu1.16.04.5) over (1.9-3.2ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up vlan (1.9-3.2ubuntu1.16.04.5) ...
Installing new version of config file /etc/network/if-pre-up.d/vlan ...
Installing new version of config file /etc/network/if-up.d/ip ...
root@host1:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto ens160
iface ens160 inet static
address 10.12.31.211
netmask 255.255.252.0
network 10.12.28.0
broadcast 10.12.31.255
gateway 10.12.28.6
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 10.12.28.6
up route add -net 172.22.0.0 netmask 255.255.0.0 gw 10.12.28.1 ens160
auto ens192
iface ens192 inet manual
auto ens192.10
iface ens192.10 inet manual
vlan-raw-device ens192
auto ens192.20
iface ens192.20 inet manual
vlan-raw-device ens192
root@host1:~# ifup ens192.10
WARNING: Could not open /proc/net/vlan/config. Maybe you need to load the 8021q module, or maybe you are not using PROCFS??
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 10 to IF -:ens192:-
ifquery: recursion detected for interface ens192 in parent-lock phase
ifquery: recursion detected for parent interface ens192 in parent-lock phase
ifquery: recursion detected for parent interface ens192 in parent-lock phase
root@host1:~# ifup ens192.20
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 20 to IF -:ens192:-
ifquery: recursion detected for interface ens192 in parent-lock phase
ifquery: recursion detected for parent interface ens192 in parent-lock phase
ifquery: recursion detected for parent interface ens192 in parent-lock phase
root@host1:~# cat /proc/net/vlan/config
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
ens192.10 | 10 | ens192
ens192.20 | 20 | ens192
root@host1:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:4c:70 brd ff:ff:ff:ff:ff:ff
3: ens192: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:22:32 brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:a2:9d:f7:13 brd ff:ff:ff:ff:ff:ff
7: ens192.10@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:22:32 brd ff:ff:ff:ff:ff:ff
8: ens192.20@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:22:32 brd ff:ff:ff:ff:ff:ff
root@host1:~# docker network create -d macvlan --subnet 172.16.10.0/24 --gateway 172.16.10.1 -o parent=ens192.10 mac_net10
884e50ddfb92c2454b4e597e6beeaf1f1f2d4f6196314d900f20c40f0d0a0c78
root@host1:~# docker network create -d macvlan --subnet 172.16.20.0/24 --gateway 172.16.20.1 -o parent=ens192.20 mac_net20
c402380a197da23fa5537fa3a36b5a82fcf30d3b999a48bda4fe82b69861b6dd
root@host1:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
9e26e05efc49 bridge bridge local
bb03f7574aa2 host host local
d60df792c936 mac_net1 macvlan local
884e50ddfb92 mac_net10 macvlan local
c402380a197d mac_net20 macvlan local
11e39328a6d1 none null local
root@host1:~# docker run -itd --name bbox_10_1 --ip 172.16.10.101 --network mac_net10 busybox
3cbcdbce63eb19024ca436fea761a4e6e154a6e7cbe26b9d6c50767dcb783026
root@host1:~# docker run -itd --name bbox_20_1 --ip 172.16.20.201 --network mac_net20 busybox
a9b648d4599a58efc64ad29db5dc484713d80803642e26910e09fcfefa54fab7
root@host1:~# docker exec bbox_10_1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:10:0a:65 brd ff:ff:ff:ff:ff:ff
root@host1:~# docker exec bbox_20_1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:10:14:c9 brd ff:ff:ff:ff:ff:ff
在host2 上做同样的操作
root@host2:~# apt-get install vlan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
vlan
1 upgraded, 0 newly installed, 0 to remove and 125 not upgraded.
Need to get 30.7 kB of archives.
After this operation, 45.1 kB disk space will be freed.
Get:1 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 vlan amd64 1.9-3.2ubuntu1.16.04.5 [30.7 kB]
Fetched 30.7 kB in 0s (393 kB/s)
(Reading database ... 60147 files and directories currently installed.)
Preparing to unpack .../vlan_1.9-3.2ubuntu1.16.04.5_amd64.deb ...
Unpacking vlan (1.9-3.2ubuntu1.16.04.5) over (1.9-3.2ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up vlan (1.9-3.2ubuntu1.16.04.5) ...
Installing new version of config file /etc/network/if-pre-up.d/vlan ...
Installing new version of config file /etc/network/if-up.d/ip ...
root@host2:~# apt-get install vlan
Reading package lists... Done
Building dependency tree
Reading state information... Done
vlan is already the newest version (1.9-3.2ubuntu1.16.04.5).
0 upgraded, 0 newly installed, 0 to remove and 125 not upgraded.
root@host2:~# vim /etc/network/interfaces
root@host2:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto ens160
iface ens160 inet static
address 10.12.31.212
netmask 255.255.252.0
network 10.12.28.0
broadcast 10.12.31.255
gateway 10.12.28.6
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 10.12.28.6
up route add -net 172.22.0.0 netmask 255.255.0.0 gw 10.12.28.1 ens160
uto ens192
iface ens192 inet manual
auto ens192.10
iface ens192.10 inet manual
vlan-raw-device ens192
auto ens192.20
iface ens192.20 inet manual
vlan-raw-device ens192
root@host2:~# ifup ens192.10
WARNING: Could not open /proc/net/vlan/config. Maybe you need to load the 8021q module, or maybe you are not using PROCFS??
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 10 to IF -:ens192:-
ifquery: recursion detected for parent interface ens192 in parent-lock phase
ifquery: recursion detected for parent interface ens192 in parent-lock phase
root@host2:~# ifup ens192.20
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 20 to IF -:ens192:-
ifquery: recursion detected for parent interface ens192 in parent-lock phase
ifquery: recursion detected for parent interface ens192 in parent-lock phase
root@host2:~# cat /proc/net/vlan/config
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
ens192.10 | 10 | ens192
ens192.20 | 20 | ens192
root@host2:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:13:59 brd ff:ff:ff:ff:ff:ff
3: ens192: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:1b:c0 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:6c:e4:0d:c1 brd ff:ff:ff:ff:ff:ff
8: ens192.10@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:1b:c0 brd ff:ff:ff:ff:ff:ff
9: ens192.20@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:87:1b:c0 brd ff:ff:ff:ff:ff:ff
root@host2:~# docker network create -d macvlan --subnet 172.16.10.0/24 --gateway 172.16.10.1 -o parent=ens192.10 mac_net10
a90d23d941a9e16332546375cb6b4c00ca3002315bb808a27c683b30ca6b46b0
root@host2:~# docker network create -d macvlan --subnet 172.16.20.0/24 --gateway 172.16.20.1 -o parent=ens192.20 mac_net20
d7312840540387493e70f3d9eb3c136f8e76f51ccc4af9b9913fb2e8765b8f98
root@host2:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
65563241b1ff bridge bridge local
cf4c89650a1f host host local
39f1aab9f5b8 mac_net1 macvlan local
a90d23d941a9 mac_net10 macvlan local
d73128405403 mac_net20 macvlan local
2f7d79e0114d none null local
root@host2:~# docker run -itd --name bbox_10_2 --ip 172.16.10.102 --network mac_net10 busybox
97be9c3ca95c3a68852bb6f20b04f6b603903140f8b24c56ce7def4dc49d672e
root@host2:~# docker run -itd --name bbox_20_2 --ip 172.16.20.202 --network mac_net20 busybox
652af91246d04263826933ba8e2334c363863ea263b6289b934d15b5193c89ef
root@host2:~# docker exec bbox_10_2 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:10:0a:66 brd ff:ff:ff:ff:ff:ff
root@host2:~# docker exec bbox_20_2 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
11: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:10:14:ca brd ff:ff:ff:ff:ff:ff
以上操作完毕后，两个host上的容器网络配置如下
root@host1:~# docker exec bbox_10_1 ip r
default via 172.16.10.1 dev eth0
172.16.10.0/24 dev eth0 scope link src 172.16.10.101
root@host1:~# docker exec bbox_20_1 ip r
default via 172.16.20.1 dev eth0
172.16.20.0/24 dev eth0 scope link src 172.16.20.201
root@host2:~# docker exec bbox_10_2 ip r
default via 172.16.10.1 dev eth0
172.16.10.0/24 dev eth0 scope link src 172.16.10.102
root@host2:~# docker exec bbox_20_2 ip r
default via 172.16.20.1 dev eth0
172.16.20.0/24 dev eth0 scope link src 172.16.20.202
最后需要注意vmware网络 需要配置vlan id 全部（4095）
