OpenStack Neutron：实验环境描述；linux bridge driver支持的网络类型（local、flat、vlan、vxlan）；网络产品详细说明（dnsmasq、floating IP、安全组、FWaaS、LBaaS）
OpenStack Neutron：网络类型示意图（local、flat、vlan、vxlan）；网络产品简介（dnsmasq、floating IP、安全组、FWaaS、LBaaS）
OpenStack 的网络分类;实验网络环境描述 ==================================================================================== OpenStack 的网络分类： Management 网络 用于节点之间 message queue 内部通信以及访问 database 服务，所有的节点都需要连接到 management 网络。 API 网络 OpenStack 各组件通过该网络向用户暴露 API 服务。Keystone, Nova, Neutron, Glance, Cinder, Horizon 的 endpoints 均配置在 API 网络上。通常，管理员也通过 API 网络 SSH 管理各个节点。 VM 网络 VM 网络也叫 tenant 网络，用于 instance 之间通信。 VM 网络可以选择的类型包括 local, flat, vlan, vxlan 和 gre。 VM 网络由 Neutron 配置和管理。 External 网络 External 网络指的是 VM 网络之外的网络，该网络不由 Neutron 管理。 Neutron 可以将 router attach 到 External 网络，为 instance 提供访问外部网络的能力。 External 网络可能是企业的 intranet，也可能是 internet。 ---------------------------------------------- Neutron网络分类： local, flat, vlan, vxlan, gre ---------------------------------------------------------------------------------- 我的实验环境采用下面的网卡分配方式： 1. 控制节点 3 网卡（ens33, ens37, ens38），计算节点 2 网卡（ens33, ens37）。 2. 合并 Management 和 API 网络，使用 ens33，IP 段为 192.168.1.0/24 3. VM 网络使用 ens37。 4. 控制节点的 ens38 与 External 网络连接，IP 段为 192.168.1.0/24。 实验网络环境描述：075 - 为 Neutron 准备物理基础设施（I） https://mp.weixin.qq.com/s?__biz=MzIwMTM5MjUwMg==&mid=2653587665&idx=1&sn=249116c3d240b6fcf03a6b927ef416dc&chksm=8d3080c8ba4709de895e84b8f5fe35fa7e537ccfb4ae765b10ac9b2de55eb13e47e5cf1b3d1e&scene=21#wechat_redirect
linux bridge测试：mechanism driver默认为open vswitch，配置使用linux bridge；linux bridge 环境，linux虚拟设备的命名规则 =========================================================================================================== 配置 linux-bridge mechanism driver 1.配置core_plugin root@ubuntu:~# cat /etc/neutron/neutron.conf |grep core_plugin #计算节点、控制节点都有该配置 core_plugin = ml2 #Q版应该是进行了修改；教程为N版，此处core_plugin的ml2名称是很长的一串名称 2.配置ml2使用mechanism driver为linux bridge Neutorn ML2 plugin 默认使用的 mechanism driver 是 open vswitch 而不是 linux bridge。 root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep mechanism_drivers mechanism_drivers = linuxbridge #控制节点和计算节点都有该配置 3.查看neutron-linuxbridge-agent进程是否已开启 stack@ubuntu:~$ ps -ef |grep linuxbridge stack 29660 1 0 09:26 ? 00:00:05 /usr/bin/python /usr/local/bin/neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini ----------------------------------------------------------------------------------------------------------------------------- linux bridge 环境，linux虚拟设备的命名规则： 在 linux bridge 环境中，一个数据包从 instance 发送到物理网卡会经过下面几个类型的设备： tap interface命名为 tapN (N 为 0, 1, 2, 3......) linux bridge命名为 brqXXXX。 vlan interface命名为 ethX.Y（X 为 interface 的序号，Y 为 vlan id） vxlan interface命名为 vxlan-Z（z 是 VNI） 物理 interface命名为 ethX（X 为 interface 的序号） vlan interface 会在 vlan 网络中使用；vxlan interface 会在 vxlan 网络中使用。linux-bridge 支持 local, flat, vlan 和 vxlan 四种 network type，目前不支持 gre。 ----------------------------------------------------------------------------------------------------------------------------- Web GUI 创建网络的2个途径： 1. Project -> Network -> Networks 2. Admin -> Network -> Networks admin创建与普通用户创建 network 不同的地方： 1.可以选择该 network 属于哪个 Project（租户）。 2.可以选择 network type。 3.可以指定 network 是否与其他 Project 共享。 4.可以指定是否为 external network。 白屏创建网络的说明： https://mp.weixin.qq.com/s?__biz=MzIwMTM5MjUwMg==&mid=2653587643&idx=1&sn=7ad568889d9f44c77d7c8e45a8fb33ff&chksm=8d3080a2ba4709b4d4f26cf38f65fda3a250f2b3eebc4bb1b7d50828107f9c80cc67055cce1a&scene=21#wechat_redirect
instance 如何获取 IP 和 MAC 地址；重启Neutron agent进程，使配置生效； ============================================================================================================================== 创建 instance 连接到租户网络，背后的逻辑： 将 instance 连接到 first_local_net 对于 instance “cirros-vm1”，Neutron 会在 subnet 中创建一个 port，分配 IP 和 MAC 地址，并将 port 分配给 cirros-vm1。 当 cirros-vm1 启动时： 1. 宿主机上的 neutron-linuxbridge-agent 会根据 port 信息创建 tap 设备，并连接到 local 网络所在的 bridge 2. 同时该 tap 会映射成 cirros-vm1 的虚拟网卡，即 virtual interface （VIF）。 #创建instance时，似乎不能指定IP，只能通过dhcp方式获取。 #port上只有mac；IP地址实际配置在dhcp server中 -------------------------------------------------------------------------------------------------------------------------------- 重启Neutron agent进程，使配置生效 土方法：通过杀进程，stack用户启动agent方式来重启进程，成功 root@ubuntu:~# ps -ef |grep neutron |grep stack |grep python stack 125594 1 0 08:18 ? 00:00:05 /usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini stack 125736 125594 0 08:18 ? 00:00:39 /usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini stack 125737 125594 0 08:18 ? 00:01:07 /usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini stack 125738 125594 0 08:18 ? 00:01:08 /usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini stack 125739 125594 0 08:18 ? 00:00:17 /usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini stack 127335 1 0 08:18 ? 00:00:22 /usr/bin/python /usr/local/bin/neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini stack 127926 1 0 08:18 ? 00:01:46 /usr/bin/python /usr/local/bin/neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini stack 128601 1 0 08:18 ? 00:00:14 /usr/bin/python /usr/local/bin/neutron-l3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent.ini stack 129192 1 0 08:18 ? 00:00:11 /usr/bin/python /usr/local/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/metadata_agent.ini stack 129609 129192 0 08:18 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/metadata_agent.ini stack 129611 129192 0 08:18 ? 00:00:00 /usr/bin/python /usr/local/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/metadata_agent.ini #使用stack用户跑以下脚本 /usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini & /usr/bin/python /usr/local/bin/neutron-linuxbridge-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini & /usr/bin/python /usr/local/bin/neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini & /usr/bin/python /usr/local/bin/neutron-l3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent.ini & /usr/bin/python /usr/local/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/metadata_agent.ini &
配置使用黑屏openstack CLI (devstack方式/RDO方式) ========================================================================================= 配置使用黑屏openstack CLI Devstack 的安装目录下有个 openrc 文件。source 该文件就可以配置 CLI 的环境变量。这里我们传入了两个参数，第一个参数是 OpenStack 用户名 admin；第二个参数是 Project 名 admin stack@ubuntu:~$ source devstack/openrc admin admin #devstack方式安装，用该方法加载后，可以使用openstack CLI stack@ubuntu:~$ neutron net-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+------+----------------------------------+---------------------------------------------------+ | id | name | tenant_id | subnets | +--------------------------------------+------+----------------------------------+---------------------------------------------------+ | 8642b715-3fb4-4f10-852a-9f05ff98e7e4 | flat | d93f10488f184c56a26e91c1c9dc8f52 | 686bf3ab-7e0c-4c84-acbe-0f343138fb56 102.0.0.0/24 | +--------------------------------------+------+----------------------------------+---------------------------------------------------+ ----------------------------------------------------------------------------------------------------- [root@controller ~]# cat admin-openrc.sh #RDO方式安装，用该方法加载后，可以使用openstack CLI export OS_USERNAME=admin export OS_PASSWORD=admin export OS_PROJECT_NAME=admin export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_DOMAIN_NAME=Default export OS_AUTH_URL=http://192.168.1.71:5000/v3 export OS_IDENTITY_API_VERSION=3
各种网络示意图及网络产品的精简说明，可见链接：https://www.cnblogs.com/AllenWoo/p/15760068.html
local network 概述；配置：enable local network；创建local网络及instance的底层变化 =========================================================================================================== local network 的特点：不会与宿主机的任何物理网卡相连，也不关联任何的 VLAN ID。 就是一台host内部的虚拟网络，无法与外部通信 local network小结： 1. 位于同一 local network 的 instance 可以通信。 2. 位于不同 local network 的 instance 无法通信。 3. 一个 local network 只能位于一个物理节点，无法跨节点。 ---------------------------------------------------------------------------------------------------------- 在 ML2 配置中 enable local network 配置文件：/etc/neutron/plugins/ml2/ml2_conf.ini root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep 'tenant_network_types\|type_drivers' tenant_network_types = vlan # 定义普通用户创建网络的默认type type_drivers = local,flat,vlan,gre,vxlan # ML2 加载所有 5 种网络的 type driver ---------------------------------------------------------------------------------------------------------- 创建local网络的底层变化： #创建local网络后，host内新增了一个bridge，且挂了一个tape9e29a83-ef，连接到dhcp server所在的network namespace root@ubuntu:~# brctl show brqf53d7c4e-d7 bridge name bridge id STP enabled interfaces brqf53d7c4e-d7 8000.ea444403865b no tape9e29a83-ef #新增的bridge设备、veth设备 root@ubuntu:~# ip -d a 18: tape9e29a83-ef@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brqf53d7c4e-d7 state UP group default qlen 1000 link/ether ea:44:44:03:86:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on 19: brqf53d7c4e-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ea:44:44:03:86:5b brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q inet6 fe80::1015:d7ff:fe59:c57e/64 scope link valid_lft forever preferred_lft forever #dhcp server所在的network namespace的veth pair设备 root@ubuntu:~# ip netns list qdhcp-f53d7c4e-d721-40c5-a7cc-27c70c3e3308 (id: 0) root@ubuntu:~# ip netns exec qdhcp-f53d7c4e-d721-40c5-a7cc-27c70c3e3308 ip -d a 2: ns-e9e29a83-ef@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:d8:b4:72 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 100.0.0.80/24 brd 100.0.0.255 scope global ns-e9e29a83-ef valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fed8:b472/64 scope link valid_lft forever preferred_lft forever ------------------------------------------------------------------------------------------------ 在local网络创建instance的底层变化： #bridge上新attach了一个tun设备 root@ubuntu:~# brctl show brqf53d7c4e-d7 bridge name bridge id STP enabled interfaces brqf53d7c4e-d7 8000.ea444403865b no tapb9e168aa-d1 tape9e29a83-ef #新增的tun设备 root@ubuntu:~# ip -d a 18: tape9e29a83-ef@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brqf53d7c4e-d7 state UP group default qlen 1000 link/ether ea:44:44:03:86:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on 19: brqf53d7c4e-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ea:44:44:03:86:5b brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q inet6 fe80::1015:d7ff:fe59:c57e/64 scope link valid_lft forever preferred_lft forever 20: tapb9e168aa-d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master brqf53d7c4e-d7 state UNKNOWN group default qlen 1000 link/ether fe:16:3e:33:f6:6e brd ff:ff:ff:ff:ff:ff promiscuity 1 tun #tun类型端口，另一端为instance bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on #查看kvm虚拟机 list root@ubuntu:~# virsh list Id Name State ---------------------------------------------------- 2 instance-00000002 running #查看 cirros-vm1 的配置 root@ubuntu:~# virsh edit instance-00000002 Select an editor. To change later, run 'select-editor'. 1. /bin/ed 2. /bin/nano <---- easiest 3. /usr/bin/vim.basic 4. /usr/bin/vim.tiny Choose 1-4 [2]: 3 Domain instance-00000002 XML configuration not changed. <interface type='bridge'> <mac address='fa:16:3e:33:f6:6e'/> #tun设备mac <source bridge='brqf53d7c4e-d7'/> <target dev='tapb9e168aa-d1'/> #tun设备名字 <model type='virtio'/> <mtu size='1500'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> ------------------------------------------------------------------------------------------------- 079 - 在 ML2 中 enable local network 080 - 创建第一个 local network（I） 081 - 创建第一个 local network（II） 082 - 将 instance 连接到 first_local_net 083 - 连接第二个 insance 到 first_local_net 084 - 创建第二个 local network 085 - 将 instance 连接到 second_local_net
flat network概述；配置enable flat network；创建flat网络、instance后，底层网络的变化 ============================================================================================================== flat network 是不带 tag 的网络，要求宿主机的物理网卡直接与 linux bridge 连接， 每个 flat network 都会独占一个物理网卡。 -------------------------------------------------------------------------------------------------------------- 在 ML2 配置中 enable flat network 在 /etc/neutron/plugins/ml2/ml2_conf.ini 设置 flat network 相关参数。 root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -PB1 'tenant_network_types|flat_networks|physical_interface_mappings' [ml2] tenant_network_types = flat #配置租户网络默认为flat; 因为 flat 网络与物理网卡一一对应，一般情况下租户网络不会采用 flat -- [ml2_type_flat] flat_networks = default #在 [ml2_type_flat] 中通过 flat_networks 定义了一个 flat 网络，label 为 “default”。 #flat_networks = flat1,flat2 #如果要创建多个 flat 网络，需要定义多个 label，用逗号隔开，当然也需要用到多个物理网卡， -- [linux_bridge] physical_interface_mappings = default:ens37 #在 [linux_bridge] 中通过 physical_interface_mappings 指明 default 对应的物理网卡为 ens37。 #physical_interface_mappings = flat1:eth1,flat2:eth2 #多个flat网络需要多张物理网卡 ###修改配置后，需要重启相应的agent进程，才能是配置生效 我的理解： label 与 ens37 的关系 label 是 flat 网络的标识，在创建 flat 时需要指定 label label 的名字可以是任意字符串，只要确保各个节点 ml2_conf.ini 中的 label 命名一致就可以了。 各个节点中 label 与物理网卡的对应关系可能不一样。 例如对于 label 为 “default” 的 flat network，节点 A 可能使用 eth1；节点 B 则可能使用 eth2 physical_interface_mappings = default:eth1 #节点A physical_interface_mappings = default:eth2 #节点B 初始化后，底层网络变化：无变化 ------------------------------------------------------------------------------------------------------------------ 创建flat网络后，底层网络变化： #在host上创建了1个bridge，并且attach了2个设备，1个tap，连接到dhcp；1个eth，就是物理网卡 root@ubuntu:~# brctl show bridge name bridge id STP enabled interfaces br-ex 8000.000000000000 no brq4fd512aa-1c 8000.000c29c91bad no ens37 tap84ff2657-4e virbr0 8000.000000000000 yes #bridge设备、veth pair设备、eth设备详情 root@ubuntu:~# ip -d a 24: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master brq4fd512aa-1c state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 1 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on inet6 fe80::7f36:c69c:5d21:ce16/64 scope link valid_lft forever preferred_lft forever 31: tap84ff2657-4e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brq4fd512aa-1c state UP group default qlen 1000 link/ether e2:17:f2:37:c7:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on 32: brq4fd512aa-1c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q inet 10.0.0.129/24 brd 10.0.0.255 scope global brq4fd512aa-1c valid_lft forever preferred_lft forever inet6 fe80::401b:1fff:fec0:454d/64 scope link valid_lft forever preferred_lft forever #dhcp server所在network namespace详情 root@ubuntu:~# ip netns list qdhcp-4fd512aa-1c63-4642-b9c5-c73acfead3c1 (id: 0) root@ubuntu:~# ip netns exec qdhcp-4fd512aa-1c63-4642-b9c5-c73acfead3c1 ip -d a 2: ns-84ff2657-4e@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:19:ac:74 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 102.0.0.2/24 brd 102.0.0.255 scope global ns-84ff2657-4e valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe19:ac74/64 scope link valid_lft forever preferred_lft forever ------------------------------------------------------------------------------------------------------------------ 创建instance后，底层网络变化： #bridge新增一个port，连接到instance root@ubuntu:~# brctl show bridge name bridge id STP enabled interfaces br-ex 8000.000000000000 no brq4fd512aa-1c 8000.000c29c91bad no ens37 tap84ff2657-4e tap8a019b11-d5 #新port virbr0 8000.000000000000 yes root@ubuntu:~# ip -d a 24: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master brq4fd512aa-1c state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 1 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on inet6 fe80::7f36:c69c:5d21:ce16/64 scope link valid_lft forever preferred_lft forever 31: tap84ff2657-4e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brq4fd512aa-1c state UP group default qlen 1000 link/ether e2:17:f2:37:c7:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on 32: brq4fd512aa-1c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q inet 10.0.0.129/24 brd 10.0.0.255 scope global brq4fd512aa-1c valid_lft forever preferred_lft forever inet6 fe80::401b:1fff:fec0:454d/64 scope link valid_lft forever preferred_lft forever 33: tap8a019b11-d5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master brq4fd512aa-1c state UNKNOWN group default qlen 1000 link/ether fe:16:3e:38:fc:fc brd ff:ff:ff:ff:ff:ff promiscuity 1 tun bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on inet6 fe80::fc16:3eff:fe38:fcfc/64 scope link valid_lft forever preferred_lft forever #####此处遇到了一个bug：虚拟instance获取到了非flat网段的IP。 原因描述： 我放行了安全组的所有协议和报文；同时flat网络配置了一个dhcp server； 该flat网络占用了物理网卡，连接到了VMware的虚拟网络中，该VMware虚拟网络中也配置了一个dhcp server 所以该flat网络相当于存在双dhcp server。VMware虚拟网络的dhcp server响应更快，结果虚拟instance获取了VMware虚拟网络的IP ------------------------------------------------------------------------------------------------------------------ 086 - flat network 原理与配置 087 - 创建 flat network 088 - 将 instance 连接到 flat_net
Vlan Network概述；配置enable Vlan Network；创建vlan网络、instance后，底层网络的变化 ============================================================================================================= Neutron Vlan Network 原理 vlan network 是带 tag 的网络，是实际应用最广泛的网络类型。 本质就是使用了vlan设备嘛。。。 ------------------------------------------------------------------------------------------------------------- 在 ML2 中配置 Vlan Network root@ubuntu:/opt/stack/devstack# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -P "tenant_network_types|network_vlan_ranges|physical_interface_mappings" -B1 [ml2] tenant_network_types = vlan #租户创建的网络默认为vlan网络 -- [ml2_type_vlan] network_vlan_ranges = default_lable:2088:2098 #vlan网络的标签为default_lable；租户创建的vlan网络vlan id范围2088-2098（只有vlan id才是针对租户的） -- [linux_bridge] physical_interface_mappings = default_lable:ens37 #标签为default_lable的vlan网络绑定到物理网卡ens37，即创建的vlan设备为ens37.X ###配置后，需要重启agent进程，配置才生效 -------------------------------------------------------------------------------------------------------------- 创建vlan网络vlan1600，底层网络变化： 新增了bridge，接口tap02579aea-8b连接到dhcp namespace；接口ens37.1600为vlan设备 root@ubuntu:/opt/stack/devstack# brctl show bridge name bridge id STP enabled interfaces brqb3e3b524-c8 8000.000c29c91bad no ens37.1600 tap02579aea-8b #新增的bridge设备、veth pair设备、vlan设备 root@ubuntu:/opt/stack/devstack# ip -d a 24: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 2 inet 10.0.0.129/24 brd 10.0.0.255 scope global ens37 valid_lft forever preferred_lft forever 51: tap02579aea-8b@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brqb3e3b524-c8 state UP group default qlen 1000 link/ether 6e:b3:de:fd:43:9b brd ff:ff:ff:ff:ff:ff link-netnsid 1 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on 52: ens37.1600@ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brqb3e3b524-c8 state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 1 vlan protocol 802.1Q id 1600 <REORDER_HDR> bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on 53: brqb3e3b524-c8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0c:29:c9:1b:ad brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q inet6 fe80::788d:65ff:fed2:3860/64 scope link valid_lft forever preferred_lft forever ------------------------------------------------- 创建instance后，底层网络变化： root@ubuntu:/opt/stack/devstack# brctl show brqb3e3b524-c8 bridge name bridge id STP enabled interfaces brqb3e3b524-c8 8000.000c29c91bad no ens37.1600 tap02579aea-8b tap2ae4e683-e4 #新增tun设备tap2ae4e683-e4，连接到instance root@ubuntu:/opt/stack/devstack# ip -d a 55: tap2ae4e683-e4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master brqb3e3b524-c8 state UNKNOWN group default qlen 1000 link/ether fe:16:3e:b6:02:6c brd ff:ff:ff:ff:ff:ff promiscuity 1 tun bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on #测试：instance(IP 106.0.0.78)ping dhcp server (IP 106.0.0.2) 通； #ping 不存在的IP 106.0.0.3，成功在另一台host上抓到vlan1600的arp报文 root@ubuntu:~# tcpdump -nnvvei ens37 tcpdump: listening on ens37, link-type EN10MB (Ethernet), capture size 262144 bytes 22:22:58.193770 fa:16:3e:b6:02:6c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1600, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 106.0.0.3 tell 106.0.0.78, length 46 ###通过dhcp namespace成功ssh到instance中 root@ubuntu:~# ssh cirros@106.0.0.78 cirros@106.0.0.78's password: $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:b6:02:6c brd ff:ff:ff:ff:ff:ff inet 106.0.0.78/24 brd 106.0.0.255 scope global eth0 inet6 fe80::f816:3eff:feb6:26c/64 scope link valid_lft forever preferred_lft forever $ ip r s default via 106.0.0.1 dev eth0 #默认网关占用第一个IP，dhcp server占用第二个IP 106.0.0.0/24 dev eth0 src 106.0.0.78 -------------------------------------------------------- 092 - Neutron Vlan Network 原理 093 - 在 ML2 中配置 Vlan Network 094 - 创建第一个 vlan network "vlan100" 095 - 将 instance 连接到 vlan100 096 - 创建第二个 vlan network "vlan101" 097 - 将 instance 连接到 vlan101
vxlan network概述；配置enable vxlan network；创建vxlan网络、instance后，底层网络的变化 =================================================================================================================================== vxlan 目前 linux bridge 只支持 vxlan，不支持 gre；open vswitch 两者都支持。vxlan 与 gre 实现非常类似，而且 vxlan 用得较多 与 VLAN 相比，VXLAN 有下面几个优势： 1. 支持更多的二层网段。 VLAN 使用 12-bit 标记 VLAN ID，最多支持 4094 个 VLAN，这对大型云部署会成为瓶颈。VXLAN 的 ID （VNI 或者 VNID）则用 24-bit 标记，支持 16777216 个二层网段。 2. 能更好地利用已有的网络路径。 VLAN 使用 Spanning Tree Protocol 避免环路，这会导致有一半的网络路径被 block 掉。VXLAN 的数据包是封装到 UDP 通过三层传输和转发的，可以使用所有的路径。 3. 避免物理交换机 MAC 表耗尽。 由于采用隧道机制，TOR (Top on Rack) 交换机无需在 MAC 表中记录虚拟机的信息。 VXLAN 使用 VXLAN tunnel endpoint (VTEP) 设备处理 VXLAN 的封装和解封。 VTEP 可以由专有硬件来实现，也可以使用纯软件实现。目前比较成熟的 VTEP 软件实现包括： 1. 带 VXLAN 内核模块的 Linux 2. Open vSwitch Linux 如何支持 VXLAN： 1. Linux vxlan 创建一个 UDP Socket，默认在 8472 端口监听。###8472 端口 2. Linux vxlan 在 UDP socket 上接收到 vxlan 包后，解包，然后根据其中的 vxlan ID 将它转给某个 vxlan interface，然后再通过它所连接的 linux bridge 转给虚机。 3. Linux vxlan 在收到虚机发来的数据包后，将其封装为多播 UDP 包，从网卡发出。 -------------------------------------------------------------------------------------------------------------------------------- 在 ML2 中配置enable VXLAN network root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -P "tenant_network_types|mechanism_drivers|vni_ranges|local_ip" -C3 [ml2] tenant_network_types = vxlan #租户网络默认为vxlan extension_drivers = port_security mechanism_drivers = linuxbridge,l2population #指定使用的mechanism_drivers -- [ml2_type_geneve] vni_ranges = 20000:20010 #注意，不要配置错误 -- [ml2_type_vxlan] vni_ranges = 20010:20020 #租户vxlan vni范围；admin用户不受限制，vni 范围为 1-16777216 -- [vxlan] local_ip = 10.0.0.129 #指定节点上用作 VTEP 的 IP 地址；此处网卡2连接的网络承载内网流量，所以此处需要指定为网卡2的IP（网卡2的IP需要提前准备，例如手工配置；因为neutron不会帮我们配置这个IP） enable_vxlan = True ###配置后，需要重启neutron agent才能使配置生效 -------------------------------------------------------------------------------------------------------------------------------- 创建vxlan后，底层网络变化： #创建了bridge，挂了一个dhcp server + vxlan设备vxlan-20016 root@ubuntu:~# brctl show brq7cadf8a4-b7 bridge name bridge id STP enabled interfaces brq7cadf8a4-b7 8000.ea838783cb7d no tap26aab43d-5f #连接到dhcp server vxlan-20016 #vxlan设备 #新增的bridge设备 root@ubuntu:~# ip -d a show brq7cadf8a4-b7 73: brq7cadf8a4-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether ea:83:87:83:cb:7d brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q inet6 fe80::5457:afff:fe5b:8a27/64 scope link valid_lft forever preferred_lft forever #新增的veth pair 设备 root@ubuntu:~# ip -d a show tap26aab43d-5f 71: tap26aab43d-5f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq7cadf8a4-b7 state UP group default qlen 1000 link/ether ea:83:87:83:cb:7d brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on #新增的vxlan设备 root@ubuntu:~# ip -d a show vxlan-20016 72: vxlan-20016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq7cadf8a4-b7 state UNKNOWN group default qlen 1000 link/ether fa:e6:1d:b1:1b:2a brd ff:ff:ff:ff:ff:ff promiscuity 1 vxlan id 20016 group 224.0.0.1 dev ens37 srcport 0 0 dstport 8472 ageing 300 udpcsum bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on #新增的vxlan设备 root@ubuntu:~# ip -d l show vxlan-20016 72: vxlan-20016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq7cadf8a4-b7 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fa:e6:1d:b1:1b:2a brd ff:ff:ff:ff:ff:ff promiscuity 1 vxlan id 20016 group 224.0.0.1 dev ens37 srcport 0 0 dstport 8472 ageing 300 udpcsum bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on addrgenmode none -------------------------------------- 创建instance后，底层网络变化： root@ubuntu:~# brctl show brq7cadf8a4-b7 bridge name bridge id STP enabled interfaces brq7cadf8a4-b7 8000.ea838783cb7d no tap26aab43d-5f tap457326a9-f2 #新增tun接口，连接到instance vxlan-20016 ##新增tun接口，连接到instance root@ubuntu:~# ip -d a show tap457326a9-f2 74: tap457326a9-f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master brq7cadf8a4-b7 state UNKNOWN group default qlen 1000 link/ether fe:16:3e:ae:52:5f brd ff:ff:ff:ff:ff:ff promiscuity 1 tun bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on #vxlan instance ping一个不存在的IP，居然抓到了arp广播包，该arp包封装为vxlan组播包。。。 root@ubuntu:~# tcpdump -nnvvei ens37 -c1 #在物理网卡抓包 tcpdump: listening on ens37, link-type EN10MB (Ethernet), capture size 262144 bytes 12:19:57.378329 00:0c:29:c9:1b:ad > 01:00:5e:00:00:01, ethertype IPv4 (0x0800), length 92: (tos 0x0, ttl 1, id 8446, offset 0, flags [none], proto UDP (17), length 78) 10.0.0.129.50041 > 224.0.0.1.8472: [bad udp cksum 0xeacd -> 0xef8c!] OTV, flags [I] (0x08), overlay 0, instance 20016 fa:16:3e:ae:52:5f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 110.0.0.27 tell 110.0.0.26, length 28 root@ubuntu:~# tcpdump -nnvvei vxlan-20016 -c5 #在vxlan设备抓包 tcpdump: listening on vxlan-20016, link-type EN10MB (Ethernet), capture size 262144 bytes 12:23:44.480722 fa:16:3e:ae:52:5f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 110.0.0.27 tell 110.0.0.26, length 28 12:23:45.480333 fa:16:3e:ae:52:5f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 110.0.0.27 tell 110.0.0.26, length 28 12:23:46.481414 fa:16:3e:ae:52:5f > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 110.0.0.27 tell 110.0.0.26, length 28 #########################host间vxlan设备的ping包还未验证 -------------------------------------------------------------------- 108 - VXLAN 概念（Part I） 109 - VXLAN 概念（Part II） 110 - 在 ML2 中配置 VXLAN 111 - 创建 VXLAN 112 - 部署 instance 到 VXLAN
L2 Population;配置enable l2population =========================================================================================================================== L2 Population 原理 是用来提高 VXLAN 网络 Scalability 的。 没有L2 Population，那么vm发出的广播报需要在vxlan全网内泛洪，开销很大；当设备集群增大，vxlan vtep节点增多时，这种泛洪的影响迅速加大，影响性能。 L2 Population 的作用是在 VTEP 上提供 Porxy ARP 功能，使得 VTEP 能够预先获知 VXLAN 网络中如下信息： 1. VM IP -- MAC 对应关系 2. VM -- VTEP 的对应关系 当 VM A 需要与 VM G 通信时： 1. Host 1 上的 VTEP 直接响应 VM A 的 APR 请求，告之 VM G 的 MAC 地址。 2. 因为 Host 1 上的 VTEP 知道 VM G 位于 Host 4，会将封装好的 VXLAN 数据包直接发送给 Host 4 的 VTEP。 这样就解决了 MAC 地址学习和 APR 广播的问题，从而保证了 VXLAN 的 Scalability VTEP 是如何提前获知 IP -- MAC -- VTEP 相关信息的呢？ 1. Neutron 知道每一个 port 的状态和信息； port 保存了 IP，MAC 相关数据。 2. instance 启动时，其 port 状态变化过程为：down -> build -> active。 3. 每当 port 状态发生变化时，Neutron 都会通过 RPC 消息通知各节点上的 Neutron agent，使得 VTEP 能够更新 VM 和 port 的相关信息。 Neutron agent-------L2 Population------Porxy ARP 功能 目前 L2 Population 支持 VXLAN with Linux bridge 和 VXLAN/GRE with OVS。 ------------------------------------------------------------------------------------------------------------------------------ 配置l2population root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -P "tenant_network_types|mechanism_drivers|vni_ranges|local_ip|l2_population" -C3 [ml2] tenant_network_types = vxlan #客户网络默认为vxlan网络 extension_drivers = port_security mechanism_drivers = linuxbridge,l2population #配置l2population -- [ml2_type_vxlan] vni_ranges = 20010:20020 -- [vxlan] enable_vxlan = True l2_population = True #使能l2_population local_ip = 10.0.0.129 ------------------------------------------------------------------------------------------------------------------------------ 配置L2 Population 前： root@ubuntu:~# ip -d a show vxlan-20016 72: vxlan-20016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq7cadf8a4-b7 state UNKNOWN group default qlen 1000 link/ether fa:e6:1d:b1:1b:2a brd ff:ff:ff:ff:ff:ff promiscuity 1 vxlan id 20016 group 224.0.0.1 dev ens37 srcport 0 0 dstport 8472 ageing 300 udpcsum bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on root@ubuntu:~# bridge fdb show dev vxlan-20016 #查看forwarding database fa:e6:1d:b1:1b:2a vlan 1 master brq7cadf8a4-b7 permanent fa:e6:1d:b1:1b:2a master brq7cadf8a4-b7 permanent 00:00:00:00:00:00 dst 224.0.0.1 via ens37 self permanent 配置L2 Population 后：（配置前后无变化，测试失败了） root@ubuntu:~# ip -d a show vxlan-20016 72: vxlan-20016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq7cadf8a4-b7 state UNKNOWN group default qlen 1000 link/ether fa:e6:1d:b1:1b:2a brd ff:ff:ff:ff:ff:ff promiscuity 1 vxlan id 20016 group 224.0.0.1 dev ens37 srcport 0 0 dstport 8472 ageing 300 udpcsum #L2 Population 生效后，创建的 vxlan-20016 会多一个 Proxy ARP 功能，未体现出来？？？？ bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on root@ubuntu:~# bridge fdb show dev vxlan-20016 fa:e6:1d:b1:1b:2a vlan 1 master brq7cadf8a4-b7 permanent fa:e6:1d:b1:1b:2a master brq7cadf8a4-b7 permanent 00:00:00:00:00:00 dst 224.0.0.1 via ens37 self permanent # bridge fdb 中保存了目标mac与VTEP的关系，即根据目标mac，将数据包直接发往目标VTEP。这个也未体现出来？？？ # 试验中没有看到教程中预期的结果，可能的原因：1.L2 Population实际未生效；2.因为没有实际建立vxlan隧道（因为计算节点的注册失败了） -------------------------------------------------------------------- 113 - L2 Population 原理 114 - 配置 L2 Population
vxlan及L2 Population的一些跨host现象由于环境部署原因导致没看到现象，需要重做！！！！！
DHCP agent、dnsmasq软件；配置文件、日志信息；dnsmasq软件的部署---network namespace ============================================================================================================================ Neutron 提供 DHCP 服务的组件是 DHCP agent。DHCP agent 在网络节点运行上，默认通过 dnsmasq 实现 DHCP 功能。 DHCP agent 会为每个 network 创建一个目录 /opt/stack/data/neutron/dhcp/，用于存放该 network 的 dnsmasq 配置文件。 dnsmasq 是一个提供 DHCP 和 DNS 服务的开源软件。dnsmasq 与 network 是一对一关系，一个 dnsmasq 进程可以为同一 netowrk 中所有 enable 了 DHCP 的 subnet 提供服务。 DHCP agent 的配置文件/etc/neutron/dhcp_agent.ini root@ubuntu:~# cat /etc/neutron/dhcp_agent.ini |grep -P "interface_driver|dhcp_driver" interface_driver = linuxbridge #使用 linux bridge 连接 DHCP namespace interface。 dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq #使用 dnsmasq 实现 DHCP。 dnsmasq 重要的启动参数： --dhcp-hostsfile 存放 DHCP host 信息的文件，这里的 host 在我们这里实际上就是 instance。dnsmasq 从该文件获取 host 的 IP 与 MAC 的对应关系。每个 host 对应一个条目，信息来源于 Neutron 数据库。 --interface 指定提供 DHCP 服务的 interface。dnsmasq 会在该 interface 上监听 instance 的 DHCP 请求。 interface 是 ns-19a0ed3d-fe；bridge上的接口为DHCP interface 叫 tap19a0ed3d-fe 从名称上看，ns-19a0ed3d-fe 和 tap19a0ed3d-fe 应该存在某种联系：tap19a0ed3d-fe 与 ns-19a0ed3d-fe 就是一对 veth pair dnsmasq是部署在Linux Network Namespace中的，也就是说通过network namespace完成隔离 stack@ubuntu:~$ ps -ef |grep 9f05ff98e7e4 nobody 59975 1 0 16:42 ? 00:00:00 dnsmasq --no-hosts --pid-file=/opt/stack/data/neutron/dhcp/8642b715-3fb4-4f10-852a-9f05ff98e7e4/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/8642b715-3fb4-4f10-852a-9f05ff98e7e4/host --addn-hosts=/opt/stack/data/neutron/dhcp/8642b715-3fb4-4f10-852a-9f05ff98e7e4/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/8642b715-3fb4-4f10-852a-9f05ff98e7e4/opts --dhcp-leasefile=/opt/stack/data/neutron/dhcp/8642b715-3fb4-4f10-852a-9f05ff98e7e4/leases --dhcp-match=set:ipxe,175 --local-service --bind-dynamic --dhcp-range=set:subnet-686bf3ab-7e0c-4c84-acbe-0f343138fb56,102.0.0.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=256 --conf-file= --domain=openstacklocal #该文件中保存了VM分配的IP stack@ubuntu:~$ cat /opt/stack/data/neutron/dhcp/8642b715-3fb4-4f10-852a-9f05ff98e7e4/host fa:16:3e:b5:73:0d,host-102-0-0-251.openstacklocal,102.0.0.251 ------------------------------------------------------------------------------------------------------------------ dnsmasq 默认将日志记录到 /var/log/syslog stack@ubuntu:~$ sudo ip netns exec qdhcp-8642b715-3fb4-4f10-852a-9f05ff98e7e4 ls -l /var/log/syslog -rw-r----- 1 syslog adm 102379947 1月 2 21:21 /var/log/syslog #namespace中的syslog文件其实就是host的syslog文件 stack@ubuntu:~$ ll /var/log/syslog -rw-r----- 1 syslog adm 102386633 1月 2 21:21 /var/log/syslog #这是一个综合的日志文件，很多内容。。。 stack@ubuntu:~$ sudo cat /var/log/syslog |grep fa:16:3e:b5:73:0d |grep dnsmasq-dhcp #dhcp报文交互日志 Jan 2 21:14:53 ubuntu dnsmasq-dhcp[59975]: DHCPDISCOVER(ns-af361670-dd) fa:16:3e:b5:73:0d Jan 2 21:14:53 ubuntu dnsmasq-dhcp[59975]: DHCPOFFER(ns-af361670-dd) 102.0.0.251 fa:16:3e:b5:73:0d Jan 2 21:14:53 ubuntu dnsmasq-dhcp[59975]: DHCPREQUEST(ns-af361670-dd) 102.0.0.251 fa:16:3e:b5:73:0d Jan 2 21:14:53 ubuntu dnsmasq-dhcp[59975]: DHCPACK(ns-af361670-dd) 102.0.0.251 fa:16:3e:b5:73:0d host-102-0-0-251 instance 获取 IP 的过程如下： 1. cirros-vm1 开机启动，发出 DHCPDISCOVER 广播，该广播消息在整个 flat_net 中都可以被收到。 2. 广播到达 veth tap19a0ed3d-fe，然后传送给 veth pair 的另一端 ns-19a0ed3d-fe。dnsmasq 在它上面监听，dnsmasq 检查其 host 文件，发现有对应项，于是dnsmasq 以 DHCPOFFER 消息将 IP（172.16.1.103）、子网掩码（255.255.255.0）、地址租用期限等信息发送给 cirros-vm1。 3. cirros-vm1 发送 DHCPREQUEST 消息确认接受此 DHCPOFFER。 4. dnsmasq 发送确认消息 DHCPACK，整个过程结束。 ======================================================================================================================= namespace解析： 在二层网络上，VLAN 可以将一个物理交换机分割成几个独立的虚拟交换机。 类似地，在三层网络上，Linux network namespace 可以将一个物理三层网络分割成几个独立的虚拟三层网络。 每个 dnsmasq 进程都位于独立的 namespace, 命名为 qdhcp-<network id> 使用命令neutron net-list ip netns list 命令列出所有的 namespace。 #使用这2个命令查看到的namespace名称不一样，但是存在关联 宿主机本身也有一个 namespace，叫 root namespace，拥有所有物理和虚拟 interface device。物理 interface 只能位于 root namespace。 新创建的 namespace 默认只有一个 loopback device。管理员可以将虚拟 interface，例如 bridge，tap 等设备添加到某个 namespace。 tap19a0ed3d-fe 与 ns-19a0ed3d-fe 就是一对 veth pair #所以连接到ns的根本就不是tap设备，只是为了方便识别，其实是veth pair设备
routing功能：物理router、虚拟router；enable虚拟router；创建虚拟router、打通网络后，底层网络的变化；使用namespace实现虚拟router，达到网络重叠的目的； ======================================================================================================== 打破vlan网络间的隔离：routing 路由服务（Routing）提供跨 subnet 联通功能。可以是物理 router 或者虚拟 router。 使用物理router，就是将router接入到网络中，并在router上配置vlan网络网关 如果要使用虚拟 router，需要启用 L3 agent。L3 agent 会在控制节点或者网络节点上运行虚拟 router，为 subnet 提供路由服务。 -------------------------------------------------------------------------------------------------------- 配置虚拟 router Neutron 的路由服务是由 l3 agent 提供的。 除此之外，l3 agent 通过 iptables 提供 firewall 和 floating ip 服务。 root@ubuntu:~# cat /etc/neutron/l3_agent.ini |grep interface_driver #配置文件，位于控制节点或网络节点上 interface_driver = linuxbridge #使用linux bridge(Q版) ###interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver #使用linux bridge（N版） ###interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver #选用 open vswitch -------------------------------------------------------------------------------------------------------- 创建虚拟router后，底层网络无变化 ip netns #查看无变化 neutron router-list #可以查看到虚拟router ###此处的租户网络间打通是由vlan网络演示（在linux bridge driver下，都是租户网络bridge连接到虚拟router，和租户的网络类型无关（例如vxlan和vlan）） 创建虚拟router接口，即使用veth pair连接虚拟router和vlan网络bridge后,底层网络变化： #从部署经验来看，只有等真正地需要配置活动的接口or虚拟设备，底层网络才可以看到变化；只是创建虚拟router，创建一个不分配dhcp的租户网络，在底层都看不到变化 #新增的虚拟router namespace qrouter-xxxxxxx #虚拟router新增的port默认IP为租户网络的网关 root@ubuntu:~# ip netns qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 (id: 2) qdhcp-b3e3b524-c8c2-4be8-8568-23670b6464f4 (id: 1) qdhcp-04b2c4ca-0bd0-4e81-8a99-d81152e510c4 (id: 0) #可以发现qr-19c418e2-0b和tap19c418e2-0b是一对veth pair；而brqb3e3b524-c8则是vlan 1600网络的bridge root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 ip a 2: qr-19c418e2-0b@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:bb:0f:df brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 106.0.0.1/24 brd 106.0.0.255 scope global qr-19c418e2-0b #自动配置了vlan网络的网关 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:febb:fdf/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~# brctl show brqb3e3b524-c8 bridge name bridge id STP enabled interfaces brqb3e3b524-c8 8000.000c29c91bad no ens37.1600 tap02579aea-8b tap19c418e2-0b tap2ae4e683-e4 root@ubuntu:~# ip -d a show tap19c418e2-0b 56: tap19c418e2-0b@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brqb3e3b524-c8 state UP group default qlen 1000 link/ether ee:88:0d:00:eb:b7 brd ff:ff:ff:ff:ff:ff link-netnsid 2 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on 2个vlan网络都连接到该虚拟router后，实测ping包已通。 ----------------------------------------------------------- 使用namespace来实现虚拟router的原因： 使用network namespace来实现虚拟router是为了实现网络隔离，达到网络重叠的目的； 而使用root namespace，虽然也可以完成通信，但是网络无法隔离，也就无法完成网络重叠。 ----------------------------------------------------------- 098 - Routing 功能概述 099 - 配置 L3 agent 100 - 创建 router 连通 subnet 101 - 虚拟 router 原理分析 102 - Why Namespace?
外网（访问公网能力）；配置enable外网；创建外网、打通租户网络和物理网络后，底层网络的变化；外网功能的原理（iptables） ========================================================================================================================================= 外网其实是就是一种特殊的flat或vlan网络，它在openstack中专门定义为外部网络，用于给其他租户网络提供访问物理网络or公网的能力 对于租户flat/vlan网络，因为这2种网络自身就是直接打通到物理网络的；所以对于这2种网络，外网只是多了一种选择 对于租户vxlan网络，因为vxlan网络对于物理网络而言是overlay，是天然隔离的。所以租户vxlan网络要访问物理网络or公网，有2种方式： 1.租户vxlan网络的流量走vxlan设备到物理网络，物理网络中存在vxlan VTEP，将vxlan报文解封装，暴露到物理网络（该方案在教程中没有提及） 2.租户vxlan网络的流量走外网，通过SNAT到达物理网络。（教程介绍的就是该方案） ###这里的外网指的是host内部的虚拟网络，具备访问物理网络or公网能力的网络，在openstack中被定义为外网的的虚拟网络；和物理网络没有关系。 ------------------------------------------------------------------------------------------------------------------------------------------ 访问外网 ML2 的配置 配置外部网络准备： #配置flat网络的外部网络（此处的flat网络和需要访问外网的网络的自身类型没关系，例如vlan网络可以通过flat外网访问外部网络） root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -P "flat_network|physical_interface_mappings|network_vlan_ranges" -B1 [ml2_type_flat] flat_networks = external #设置flat网络标签external -- [ml2_type_vlan] network_vlan_ranges = default_lable:2088:2098 #设置vlan网络标签default_lable -- [linux_bridge] physical_interface_mappings = default_lable:ens37,external:ens38 #default_lable网络使用物理网卡ens37，external网络使用物理网卡ens38 #配置vlan网络的外部网络（此处的vlan网络和需要访问外网的网络的自身类型没关系，例如flat网络可以通过vlan外网访问外部网络） root@ubuntu:~# cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep -P "network_vlan_ranges|physical_interface_mappings" -B1 [ml2_type_vlan] network_vlan_ranges = default_lable:2088:2098,external #设置vlan网络的2个标签 -- [linux_bridge] physical_interface_mappings = default_lable:ens37,external:ens38 #default_lable使用物理网卡ens37，external网络使用物理网卡ens38 ------------------------------------------------------------------------------------------------------------------------------------------ 创建外网 ext_net 创建flat类型的外网，将创建一个独占物理网卡的bridge 创建vlan类型的外网，将创建一个占用vlan设备的bridge（需要创建一个占用端口的虚拟设备后，vlan设备、bridge才会被真正创建） 操作上和普通的flat、vlan网络不同： 例如：勾选external network、不勾选创建子网 创建vlan类型的外网后，底层网络看起来无变化。 只有创建出占用端口的虚拟设备后，相应的网桥、vlan设备才会被创建出来；ip a、brctl show命令才能查看到相应的网桥、vlan设备 ###VMware环境，使用vlan外网和外部通信，设置比较麻烦，实际未进行测试 实际测试vlan网络访问外网是通过flat网络进行的。 2个步骤： 1.创建flat外部网络（需要勾选external network） 2.虚拟router设置网关，关联到flat外部网络 虚拟router自动连接到flat外部网络bridge 虚拟router接口自动获取一个flat外部网络IP 创建flat外部网络后，物理网络变化：（需要勾选external network） #flat外部网络bridge root@ubuntu:~# brctl show brq15d132d9-63 bridge name bridge id STP enabled interfaces brq15d132d9-63 8000.000c29c91bb7 no ens38 tap4e875b86-d2 tap7d2f69ca-e5 #tap4e875b86-d2设备连接到虚拟router root@ubuntu:~# ip -d a show tap4e875b86-d2 70: tap4e875b86-d2@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brq15d132d9-63 state UP group default qlen 1000 link/ether 72:a7:d1:8a:ff:5d brd ff:ff:ff:ff:ff:ff link-netnsid 2 promiscuity 1 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 ip -d a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: qr-19c418e2-0b@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:bb:0f:df brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 106.0.0.1/24 brd 106.0.0.255 scope global qr-19c418e2-0b #连接到vlan网络1 bridge valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:febb:fdf/64 scope link valid_lft forever preferred_lft forever 3: qr-a3de9995-f7@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:11:53:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 103.0.0.1/24 brd 103.0.0.255 scope global qr-a3de9995-f7 #连接到vlan网络2 bridge valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe11:5347/64 scope link valid_lft forever preferred_lft forever 8: qg-4e875b86-d2@if70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:e4:62:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth inet 192.168.1.94/24 brd 192.168.1.255 scope global qg-4e875b86-d2 #连接到flat外部网络 bridge valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee4:6243/64 scope link valid_lft forever preferred_lft forever #经过实测，vlan网络instance 106.0.0.78 成功访问百度 $ ip r s default via 106.0.0.1 dev eth0 106.0.0.0/24 dev eth0 src 106.0.0.78 $ curl www.baidu.com <!DOCTYPE html> <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html> --------------------------------------------------------------------------------------------------------------------------- 外网访问原理分析 访问外网的本质就是虚拟router连接到外网，同时虚拟router承担SNAT功能 虚拟router的接口的命名规则 如果 veth 用于连接租户网络，命名格式为 qr-xxx，比如 qr-d568ba1a-74 和 qr-e17162c5-00。 如果 veth 用于连接外部网络，命名格式为 qg-xxx，比如 qg-b8b32a88-03。 #访问外网的本质就是虚拟router连接到外网，同时虚拟router承担SNAT功能 root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-POSTROUTING -N neutron-l3-agent-PREROUTING -N neutron-l3-agent-float-snat -N neutron-l3-agent-snat -N neutron-postrouting-bottom -A PREROUTING -j neutron-l3-agent-PREROUTING -A OUTPUT -j neutron-l3-agent-OUTPUT -A POSTROUTING -j neutron-l3-agent-POSTROUTING -A POSTROUTING -j neutron-postrouting-bottom -A neutron-l3-agent-POSTROUTING ! -i qg-4e875b86-d2 ! -o qg-4e875b86-d2 -m conntrack ! --ctstate DNAT -j ACCEPT -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat -A neutron-l3-agent-snat -o qg-4e875b86-d2 -j SNAT --to-source 192.168.1.94 #SNAT配置 -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 192.168.1.94 -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain POSTROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 neutron-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-l3-agent-OUTPUT (1 references) target prot opt source destination Chain neutron-l3-agent-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! ctstate DNAT Chain neutron-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697 Chain neutron-l3-agent-float-snat (1 references) target prot opt source destination Chain neutron-l3-agent-snat (1 references) target prot opt source destination neutron-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0 SNAT all -- 0.0.0.0/0 0.0.0.0/0 to:192.168.1.94 ###SNAT配置 SNAT all -- 0.0.0.0/0 0.0.0.0/0 mark match ! 0x2/0xffff ctstate DNAT to:192.168.1.94 Chain neutron-postrouting-bottom (1 references) target prot opt source destination neutron-l3-agent-snat all -- 0.0.0.0/0 0.0.0.0/0 /* Perform source NAT on outgoing traffic. */ ----------------------------------------------------------------------------------------------------------------------------- 103 - 访问外网 ML2 的配置 104 - 创建外网 ext_net 105 - 外网访问原理分析
floating IP ：提供"公网or物理网络访问租户网络"的能力；配置floating IP后，虚拟router接口及iptables变化 ================================================================================================================== 外网访问虚拟网络的需求则可以通过 floating IP 来实现。 1. floating IP 提供静态 NAT 功能，建立外网 IP 与 instance 租户网络 IP 的一对一映射。 （这个不就是EIP嘛） 2. floating IP 是配置在 router 提供网关的外网 interface 上的，而非 instance 中。 但是这个 floating IP 是落在虚拟router的出接口上的，也是用iptables实现的 3. router 会根据通信的方向修改数据包的源或者目的地址。 1. floating IP 能够让外网直接访问租户网络中的 instance。这是通过在 router 上应用 iptalbes 的 NAT 规则实现的。 2. floating IP 是配置在 router 的外网 interface 上的，而非 instance，这一点需要特别注意。 ------------------------------------------------------------------------------------------------------------------- 配置 floating IP 测试申请了 floating IP 192.168.1.148，并绑定了vlan 1600网络的instance 106.0.0.78；（配置生效需要时间） root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 ip a show qg-4e875b86-d2 8: qg-4e875b86-d2@if70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:e4:62:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.1.94/24 brd 192.168.1.255 scope global qg-4e875b86-d2 valid_lft forever preferred_lft forever inet 192.168.1.148/32 brd 192.168.1.148 scope global qg-4e875b86-d2 #分配的floating IP是/32 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee4:6243/64 scope link valid_lft forever preferred_lft forever root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-POSTROUTING -N neutron-l3-agent-PREROUTING -N neutron-l3-agent-float-snat -N neutron-l3-agent-snat -N neutron-postrouting-bottom -A PREROUTING -j neutron-l3-agent-PREROUTING -A OUTPUT -j neutron-l3-agent-OUTPUT -A POSTROUTING -j neutron-l3-agent-POSTROUTING -A POSTROUTING -j neutron-postrouting-bottom -A neutron-l3-agent-OUTPUT -d 192.168.1.148/32 -j DNAT --to-destination 106.0.0.78 #在发送数据包时，若目标IP为192.168.1.148，则进行DNAT，目标IP转化为106.0.0.78 -A neutron-l3-agent-POSTROUTING ! -i qg-4e875b86-d2 ! -o qg-4e875b86-d2 -m conntrack ! --ctstate DNAT -j ACCEPT -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 -A neutron-l3-agent-PREROUTING -d 192.168.1.148/32 -j DNAT --to-destination 106.0.0.78 #当 router 接收到从外网发来的包，如果目的地址是 floating IP 192.168.1.148，将目的地址修改为vlan1600的instance IP 106.0.0.78。 -A neutron-l3-agent-float-snat -s 106.0.0.78/32 -j SNAT --to-source 192.168.1.148 #当 vlan1600的instance 发送数据到外网，源IP106.0.0.78 将被修改为 floating IP 192.168.1.148。 -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat -A neutron-l3-agent-snat -o qg-4e875b86-d2 -j SNAT --to-source 192.168.1.94 -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 192.168.1.94 -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat root@ubuntu:~# ip netns exec qrouter-6131a3a2-2db9-47b3-b9b8-44e051ec01e2 iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-l3-agent-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain POSTROUTING (policy ACCEPT) target prot opt source destination neutron-l3-agent-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 neutron-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-l3-agent-OUTPUT (1 references) target prot opt source destination DNAT all -- 0.0.0.0/0 192.168.1.148 to:106.0.0.78 ####NAT表OUTPUT链，在发送数据包时生效 Chain neutron-l3-agent-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! ctstate DNAT Chain neutron-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697 DNAT all -- 0.0.0.0/0 192.168.1.148 to:106.0.0.78 ####NAT表PREROUTING链，在接收数据包时生效 Chain neutron-l3-agent-float-snat (1 references) target prot opt source destination SNAT all -- 106.0.0.78 0.0.0.0/0 to:192.168.1.148 ####NAT表POSTROUTING链，在发送数据包时生效 #这一串链有点长。。。 neutron-l3-agent-float-snat--->neutron-l3-agent-snat--->neutron-postrouting-bottom--->POSTROUTING Chain neutron-l3-agent-snat (1 references) target prot opt source destination neutron-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0 SNAT all -- 0.0.0.0/0 0.0.0.0/0 to:192.168.1.94 SNAT all -- 0.0.0.0/0 0.0.0.0/0 mark match ! 0x2/0xffff ctstate DNAT to:192.168.1.94 Chain neutron-postrouting-bottom (1 references) target prot opt source destination neutron-l3-agent-snat all -- 0.0.0.0/0 0.0.0.0/0 /* Perform source NAT on outgoing traffic. */ ###经测试，ping包是通的；在host上可以抓包经过DNAT的报文 ###192.168.1.55 ping 192.168.1.148，抓包结果如下： root@ubuntu:~# tcpdump -nnvvei brqb3e3b524-c8 tcpdump: listening on brqb3e3b524-c8, link-type EN10MB (Ethernet), capture size 262144 bytes 10:20:22.692085 fa:16:3e:bb:0f:df > fa:16:3e:b6:02:6c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 30375, offset 0, flags [none], proto ICMP (1), length 60) 192.168.1.55 > 106.0.0.78: ICMP echo request, id 1, seq 13827, length 40 10:20:22.693084 fa:16:3e:b6:02:6c > fa:16:3e:bb:0f:df, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 5476, offset 0, flags [none], proto ICMP (1), length 60) 106.0.0.78 > 192.168.1.55: ICMP echo reply, id 1, seq 13827, length 40 ---------------------------------------------------- 106 - 创建 floating IP 107 - floating IP 原理分析
各类型租户网络的网络访问场景对比 ======================================================================================================================================================== 租户网络内互访（单host） 1.local网络，通过bridge直接互访； 2.flat网络，通过bridge直接互访； 3.vlan网络，通过bridge直接互访； 4.vxlan网络，通过bridge直接互访； 租户网络内互访（跨host） 1.local网络不能跨host部署 2.flat网络占用物理网卡，互访场景直接通过物理网络进行 3.vlan网络占用物理网卡的虚拟vlan设备，互访场景建立在物理网络的vlan基础上 4.vxlan网络占用物理网卡的虚拟vxlan设备，流量在host间传递时，走vxlan隧道 租户网络间互访 1.创建虚拟router，2个租户网络同时连接到该虚拟router 2.同时虚拟router作为网关，提供转发功能 租户网络访问外网 1.租户网络的网关配置在虚拟router上； 2.虚拟router连接到外网（flat或vlan） 3.虚拟router为租户网络流量做SNAT 外部网络访问租户网络 0.外部网络指的是物理网络或者物理网络之外的公网；外网（flat或vlan）则是openstack定义的外网，其本质是落在host内部的网络，只不过这个网络和外部网络打通了 1.为租户网络配置flaoting IP；flaoting IP落在虚拟router上 2.而floating IP取自外网（flat或vlan）网段， 3.外部网络和外网（flat或vlan）是打通的； 4.外部网络访问外网（flat或vlan）的flaoting IP，虚拟router通过iptables做DNAT
安全组（Security Group）和虚拟防火墙；安全组和FWaaS的区别 ============================================================================== Neutron 为 instance 提供了两种管理网络安全的方法：安全组（Security Group）和虚拟防火墙。 安全组的原理是通过 iptables 对 instance 所在计算节点的网络流量进行过滤。 安全组是针对租户网络bridge的接口进行配置iptables规则的。 租户网络bridge存在于宿主机中，安全组自然也就配置在宿主机了。 虚拟防火墙则由 Neutron Firewall as a Service（FWaaS）高级服务提供。其底层也是使用 iptables，在 Neutron Router 上对网络包进行过滤。-----虚拟防火墙应该是配置在虚拟router的namespace，应用在虚拟router的接口上？ -------------------------------------------- 安全组和FWaaS的区别： 安全组的应用对象是虚拟网卡，由 L2 Agent 实现，比如 neutron_openvswitch_agent 和 neutron_linuxbridge_agent。安全组会在计算节点上通过 iptables 规则来控制进出 instance 虚拟网卡的流量。也就是说：安全组保护的是 instance。 FWaaS 的应用对象是 router，可以在安全组之前控制外部过来的流量，但是对于同一个 subnet 内的流量不作限制。也就是说：FWaaS 保护的是 subnet。 FWaaS 和安全组做个比较。 相同点： 1. 底层都是通过 iptables 实现。 不同点： 1. FWaaS 的 iptables 规则应用在 router 上，保护整个租户网络；安全组则应用在虚拟网卡上，保护单个 instance。 FWaaS 的配置落在 虚拟router namespace中 安全组配置落在 宿主机 root namespace中 2. FWaaS 可以定义 allow 或者 deny 规则；安全组只能定义 allow 规则。
安全组 概述；宿主机root namespace iptables表 --------------------------------- 安全组： 每个 Project（租户）都有一个命名为 “default” 的默认安全组。点击菜单 Project -> Compute -> Access & Security，查看 Security Group 列表。 “default” 安全组有四条规则，其作用是：允许所有外出（Egress）的流量，但禁止所有进入（Ingress）的流量。 ipv4/6 出入各两条，共4条 在 devstack-controller 上执行 iptables-save 命令查看相关规则。 再host上看到安全组的规则？包含了VM的？ cirros-vm1 的 TAP interface 为 tap8bca5b86-23，可以看到： 1. iptables 的规则是应用在 Neutron port 上的，port 在这里是 cirros-vm1 的虚拟网卡 tap8bca5b86-23。 2. ingress 规则集中定义在命名为 “neutron-linuxbri-i8bca5b86-2” 的 chain 中。 3. egress 规则集中定义在命名为 “neutron-linuxbri-o8bca5b86-2” 的 chain 中。 安全组有以下特性： 1. 通过宿主机上 iptables 规则控制进出 instance 的流量。 2. 安全组作用在 instance 的 port 上。 3. 安全组的规则都是 allow，不能定义 deny 的规则。 4. instance 可应用多个安全组叠加使用这些安全组中的规则。 -------------------------------------------------------------------------------------------------------------------------------- 宿主机root namespace iptables表 #全量表 root@ubuntu:~# iptables-save # Generated by iptables-save v1.6.0 on Mon Jan 3 14:34:05 2022 *raw :PREROUTING ACCEPT [17836:25914806] :OUTPUT ACCEPT [10994:24055670] :neutron-linuxbri-OUTPUT - [0:0] :neutron-linuxbri-PREROUTING - [0:0] -A PREROUTING -j neutron-linuxbri-PREROUTING -A OUTPUT -j neutron-linuxbri-OUTPUT -A neutron-linuxbri-PREROUTING -m physdev --physdev-in brqb3e3b524-c8 -m comment --comment "Set zone for ae4e683-e4" -j CT --zone 4098 -A neutron-linuxbri-PREROUTING -i brqb3e3b524-c8 -m comment --comment "Set zone for ae4e683-e4" -j CT --zone 4098 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in tap2ae4e683-e4 -m comment --comment "Set zone for ae4e683-e4" -j CT --zone 4098 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in brq7cadf8a4-b7 -m comment --comment "Set zone for 57326a9-f2" -j CT --zone 4099 -A neutron-linuxbri-PREROUTING -i brq7cadf8a4-b7 -m comment --comment "Set zone for 57326a9-f2" -j CT --zone 4099 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in tap457326a9-f2 -m comment --comment "Set zone for 57326a9-f2" -j CT --zone 4099 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in brq7cadf8a4-b7 -m comment --comment "Set zone for ca6dc60-e1" -j CT --zone 4099 -A neutron-linuxbri-PREROUTING -i brq7cadf8a4-b7 -m comment --comment "Set zone for ca6dc60-e1" -j CT --zone 4099 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in tap4ca6dc60-e1 -m comment --comment "Set zone for ca6dc60-e1" -j CT --zone 4099 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in brqb3e3b524-c8 -m comment --comment "Set zone for 9543d91-e7" -j CT --zone 4098 -A neutron-linuxbri-PREROUTING -i brqb3e3b524-c8 -m comment --comment "Set zone for 9543d91-e7" -j CT --zone 4098 -A neutron-linuxbri-PREROUTING -m physdev --physdev-in tapb9543d91-e7 -m comment --comment "Set zone for 9543d91-e7" -j CT --zone 4098 COMMIT # Completed on Mon Jan 3 14:34:05 2022 # Generated by iptables-save v1.6.0 on Mon Jan 3 14:34:05 2022 *mangle :PREROUTING ACCEPT [18887897:14763853805] :INPUT ACCEPT [4787273:2408109277] :FORWARD ACCEPT [369541:136613100] :OUTPUT ACCEPT [4751272:2156792494] :POSTROUTING ACCEPT [5118554:2292892666] -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill COMMIT # Completed on Mon Jan 3 14:34:05 2022 # Generated by iptables-save v1.6.0 on Mon Jan 3 14:34:05 2022 *nat :PREROUTING ACCEPT [13583348:12199212739] :INPUT ACCEPT [5451:545672] :OUTPUT ACCEPT [28517:1738679] :POSTROUTING ACCEPT [30341:2117460] -A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN -A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE -A POSTROUTING -s 172.24.4.0/24 -o ens33 -j MASQUERADE COMMIT # Completed on Mon Jan 3 14:34:05 2022 # Generated by iptables-save v1.6.0 on Mon Jan 3 14:34:05 2022 *filter :INPUT ACCEPT [11011:24042482] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [11003:24057166] :neutron-filter-top - [0:0] :neutron-linuxbri-FORWARD - [0:0] :neutron-linuxbri-INPUT - [0:0] :neutron-linuxbri-OUTPUT - [0:0] :neutron-linuxbri-i2ae4e683-e - [0:0] :neutron-linuxbri-i457326a9-f - [0:0] :neutron-linuxbri-i4ca6dc60-e - [0:0] :neutron-linuxbri-ib9543d91-e - [0:0] :neutron-linuxbri-local - [0:0] :neutron-linuxbri-o2ae4e683-e - [0:0] :neutron-linuxbri-o457326a9-f - [0:0] :neutron-linuxbri-o4ca6dc60-e - [0:0] :neutron-linuxbri-ob9543d91-e - [0:0] :neutron-linuxbri-s2ae4e683-e - [0:0] :neutron-linuxbri-s457326a9-f - [0:0] :neutron-linuxbri-s4ca6dc60-e - [0:0] :neutron-linuxbri-sb9543d91-e - [0:0] :neutron-linuxbri-sg-chain - [0:0] :neutron-linuxbri-sg-fallback - [0:0] -A INPUT -j neutron-linuxbri-INPUT -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -j neutron-filter-top -A FORWARD -j neutron-linuxbri-FORWARD -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A OUTPUT -j neutron-filter-top -A OUTPUT -j neutron-linuxbri-OUTPUT -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT -A neutron-filter-top -j neutron-linuxbri-local -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap457326a9-f2 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap457326a9-f2 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap4ca6dc60-e1 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap4ca6dc60-e1 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-out tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-in tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap02579aea-8b --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap02579aea-8b --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap7d2f69ca-e5 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap7d2f69ca-e5 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap4e875b86-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap4e875b86-d2 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap19c418e2-0b --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap19c418e2-0b --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap26aab43d-5f --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap26aab43d-5f --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap4e205cd3-38 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap4e205cd3-38 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap7521868f-ee --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap7521868f-ee --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-INPUT -m physdev --physdev-in tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-linuxbri-o2ae4e683-e -A neutron-linuxbri-INPUT -m physdev --physdev-in tap457326a9-f2 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-linuxbri-o457326a9-f -A neutron-linuxbri-INPUT -m physdev --physdev-in tap4ca6dc60-e1 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-linuxbri-o4ca6dc60-e -A neutron-linuxbri-INPUT -m physdev --physdev-in tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-linuxbri-ob9543d91-e -A neutron-linuxbri-i2ae4e683-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-i2ae4e683-e -d 106.0.0.78/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i2ae4e683-e -d 255.255.255.255/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i2ae4e683-e -p udp -j RETURN -A neutron-linuxbri-i2ae4e683-e -p tcp -j RETURN -A neutron-linuxbri-i2ae4e683-e -p icmp -j RETURN -A neutron-linuxbri-i2ae4e683-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-i2ae4e683-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-i457326a9-f -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-i457326a9-f -d 110.0.0.26/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i457326a9-f -d 255.255.255.255/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i457326a9-f -p udp -j RETURN -A neutron-linuxbri-i457326a9-f -p tcp -j RETURN -A neutron-linuxbri-i457326a9-f -p icmp -j RETURN -A neutron-linuxbri-i457326a9-f -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-i457326a9-f -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-i4ca6dc60-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-i4ca6dc60-e -d 110.0.0.34/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i4ca6dc60-e -d 255.255.255.255/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i4ca6dc60-e -p udp -j RETURN -A neutron-linuxbri-i4ca6dc60-e -p tcp -j RETURN -A neutron-linuxbri-i4ca6dc60-e -p icmp -j RETURN -A neutron-linuxbri-i4ca6dc60-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-i4ca6dc60-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-ib9543d91-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-ib9543d91-e -d 106.0.0.66/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-ib9543d91-e -d 255.255.255.255/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-ib9543d91-e -p udp -j RETURN -A neutron-linuxbri-ib9543d91-e -p tcp -j RETURN -A neutron-linuxbri-ib9543d91-e -p icmp -j RETURN -A neutron-linuxbri-ib9543d91-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-ib9543d91-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-o2ae4e683-e -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o2ae4e683-e -j neutron-linuxbri-s2ae4e683-e -A neutron-linuxbri-o2ae4e683-e -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o2ae4e683-e -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-linuxbri-o2ae4e683-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-o2ae4e683-e -j RETURN -A neutron-linuxbri-o2ae4e683-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-o2ae4e683-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-o457326a9-f -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o457326a9-f -j neutron-linuxbri-s457326a9-f -A neutron-linuxbri-o457326a9-f -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o457326a9-f -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-linuxbri-o457326a9-f -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-o457326a9-f -j RETURN -A neutron-linuxbri-o457326a9-f -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-o457326a9-f -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-o4ca6dc60-e -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o4ca6dc60-e -j neutron-linuxbri-s4ca6dc60-e -A neutron-linuxbri-o4ca6dc60-e -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o4ca6dc60-e -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-linuxbri-o4ca6dc60-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-o4ca6dc60-e -j RETURN -A neutron-linuxbri-o4ca6dc60-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-o4ca6dc60-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-ob9543d91-e -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-ob9543d91-e -j neutron-linuxbri-sb9543d91-e -A neutron-linuxbri-ob9543d91-e -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-ob9543d91-e -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-linuxbri-ob9543d91-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-ob9543d91-e -j RETURN -A neutron-linuxbri-ob9543d91-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-ob9543d91-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-s2ae4e683-e -s 106.0.0.78/32 -m mac --mac-source FA:16:3E:B6:02:6C -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-linuxbri-s2ae4e683-e -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-linuxbri-s457326a9-f -s 110.0.0.26/32 -m mac --mac-source FA:16:3E:AE:52:5F -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-linuxbri-s457326a9-f -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-linuxbri-s4ca6dc60-e -s 110.0.0.34/32 -m mac --mac-source FA:16:3E:F5:53:7A -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-linuxbri-s4ca6dc60-e -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-linuxbri-sb9543d91-e -s 106.0.0.66/32 -m mac --mac-source FA:16:3E:81:53:80 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-linuxbri-sb9543d91-e -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-linuxbri-sg-chain -m physdev --physdev-out tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-i2ae4e683-e -A neutron-linuxbri-sg-chain -m physdev --physdev-in tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-o2ae4e683-e -A neutron-linuxbri-sg-chain -m physdev --physdev-out tap457326a9-f2 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-i457326a9-f -A neutron-linuxbri-sg-chain -m physdev --physdev-in tap457326a9-f2 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-o457326a9-f -A neutron-linuxbri-sg-chain -m physdev --physdev-out tap4ca6dc60-e1 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-i4ca6dc60-e -A neutron-linuxbri-sg-chain -m physdev --physdev-in tap4ca6dc60-e1 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-o4ca6dc60-e -A neutron-linuxbri-sg-chain -m physdev --physdev-out tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-ib9543d91-e -A neutron-linuxbri-sg-chain -m physdev --physdev-in tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-ob9543d91-e -A neutron-linuxbri-sg-chain -j ACCEPT -A neutron-linuxbri-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP COMMIT # Completed on Mon Jan 3 14:34:05 2022 -------------------------------------------------------------------------------------------------------------------------------------------------- #全量filter表 root@ubuntu:~# iptables -t filter -nL Chain INPUT (policy ACCEPT) target prot opt source destination neutron-linuxbri-INPUT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67 Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0 neutron-linuxbri-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0 neutron-linuxbri-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68 Chain neutron-filter-top (2 references) target prot opt source destination neutron-linuxbri-local all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-linuxbri-FORWARD (1 references) target prot opt source destination neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2ae4e683-e4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2ae4e683-e4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap457326a9-f2 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap457326a9-f2 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap4ca6dc60-e1 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap4ca6dc60-e1 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapb9543d91-e7 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapb9543d91-e7 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap02579aea-8b --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap02579aea-8b --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap7d2f69ca-e5 --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap7d2f69ca-e5 --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap4e875b86-d2 --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap4e875b86-d2 --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap19c418e2-0b --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap19c418e2-0b --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap26aab43d-5f --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap26aab43d-5f --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap4e205cd3-38 --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap4e205cd3-38 --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap7521868f-ee --physdev-is-bridged /* Accept all packets when port is trusted. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap7521868f-ee --physdev-is-bridged /* Accept all packets when port is trusted. */ Chain neutron-linuxbri-INPUT (1 references) target prot opt source destination neutron-linuxbri-o2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2ae4e683-e4 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ neutron-linuxbri-o457326a9-f all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap457326a9-f2 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ neutron-linuxbri-o4ca6dc60-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap4ca6dc60-e1 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ neutron-linuxbri-ob9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapb9543d91-e7 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ Chain neutron-linuxbri-OUTPUT (1 references) target prot opt source destination Chain neutron-linuxbri-i2ae4e683-e (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN udp -- 0.0.0.0/0 106.0.0.78 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-i457326a9-f (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN udp -- 0.0.0.0/0 110.0.0.26 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-i4ca6dc60-e (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN udp -- 0.0.0.0/0 110.0.0.34 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-ib9543d91-e (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN udp -- 0.0.0.0/0 106.0.0.66 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-local (1 references) target prot opt source destination Chain neutron-linuxbri-o2ae4e683-e (2 references) target prot opt source destination RETURN udp -- 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ neutron-linuxbri-s2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 /* Prevent DHCP Spoofing by VM. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-o457326a9-f (2 references) target prot opt source destination RETURN udp -- 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ neutron-linuxbri-s457326a9-f all -- 0.0.0.0/0 0.0.0.0/0 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 /* Prevent DHCP Spoofing by VM. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-o4ca6dc60-e (2 references) target prot opt source destination RETURN udp -- 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ neutron-linuxbri-s4ca6dc60-e all -- 0.0.0.0/0 0.0.0.0/0 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 /* Prevent DHCP Spoofing by VM. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-ob9543d91-e (2 references) target prot opt source destination RETURN udp -- 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ neutron-linuxbri-sb9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 /* Prevent DHCP Spoofing by VM. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-linuxbri-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-linuxbri-s2ae4e683-e (1 references) target prot opt source destination RETURN all -- 106.0.0.78 0.0.0.0/0 MAC FA:16:3E:B6:02:6C /* Allow traffic from defined IP/MAC pairs. */ DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Drop traffic without an IP/MAC allow rule. */ Chain neutron-linuxbri-s457326a9-f (1 references) target prot opt source destination RETURN all -- 110.0.0.26 0.0.0.0/0 MAC FA:16:3E:AE:52:5F /* Allow traffic from defined IP/MAC pairs. */ DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Drop traffic without an IP/MAC allow rule. */ Chain neutron-linuxbri-s4ca6dc60-e (1 references) target prot opt source destination RETURN all -- 110.0.0.34 0.0.0.0/0 MAC FA:16:3E:F5:53:7A /* Allow traffic from defined IP/MAC pairs. */ DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Drop traffic without an IP/MAC allow rule. */ Chain neutron-linuxbri-sb9543d91-e (1 references) target prot opt source destination RETURN all -- 106.0.0.66 0.0.0.0/0 MAC FA:16:3E:81:53:80 /* Allow traffic from defined IP/MAC pairs. */ DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Drop traffic without an IP/MAC allow rule. */ Chain neutron-linuxbri-sg-chain (8 references) target prot opt source destination neutron-linuxbri-i2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2ae4e683-e4 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-o2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2ae4e683-e4 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-i457326a9-f all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap457326a9-f2 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-o457326a9-f all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap457326a9-f2 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-i4ca6dc60-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap4ca6dc60-e1 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-o4ca6dc60-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap4ca6dc60-e1 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-ib9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapb9543d91-e7 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-ob9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapb9543d91-e7 --physdev-is-bridged /* Jump to the VM specific chain. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-linuxbri-sg-fallback (8 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Default drop rule for unmatched traffic. */ -------------------------------------------------------------------------------------------------------------------------------------------------- ###根据接口来进行过滤 root@ubuntu:~# iptables-save |grep -P "\*|2ae4e683" *raw -A neutron-linuxbri-PREROUTING -m physdev --physdev-in tap2ae4e683-e4 -m comment --comment "Set zone for ae4e683-e4" -j CT --zone 4098 *mangle *nat *filter :neutron-linuxbri-i2ae4e683-e - [0:0] :neutron-linuxbri-o2ae4e683-e - [0:0] :neutron-linuxbri-s2ae4e683-e - [0:0] -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-INPUT -m physdev --physdev-in tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-linuxbri-o2ae4e683-e -A neutron-linuxbri-i2ae4e683-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-i2ae4e683-e -d 106.0.0.78/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i2ae4e683-e -d 255.255.255.255/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-i2ae4e683-e -p udp -j RETURN -A neutron-linuxbri-i2ae4e683-e -p tcp -j RETURN -A neutron-linuxbri-i2ae4e683-e -p icmp -j RETURN -A neutron-linuxbri-i2ae4e683-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-i2ae4e683-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-o2ae4e683-e -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o2ae4e683-e -j neutron-linuxbri-s2ae4e683-e -A neutron-linuxbri-o2ae4e683-e -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-o2ae4e683-e -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-linuxbri-o2ae4e683-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-o2ae4e683-e -j RETURN -A neutron-linuxbri-o2ae4e683-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-o2ae4e683-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-s2ae4e683-e -s 106.0.0.78/32 -m mac --mac-source FA:16:3E:B6:02:6C -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-linuxbri-s2ae4e683-e -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-linuxbri-sg-chain -m physdev --physdev-out tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-i2ae4e683-e -A neutron-linuxbri-sg-chain -m physdev --physdev-in tap2ae4e683-e4 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-o2ae4e683-e root@ubuntu:~# iptables-save |grep -P "\*|b9543d91" *raw -A neutron-linuxbri-PREROUTING -m physdev --physdev-in tapb9543d91-e7 -m comment --comment "Set zone for 9543d91-e7" -j CT --zone 4098 *mangle *nat *filter :neutron-linuxbri-ib9543d91-e - [0:0] :neutron-linuxbri-ob9543d91-e - [0:0] :neutron-linuxbri-sb9543d91-e - [0:0] -A neutron-linuxbri-FORWARD -m physdev --physdev-out tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-FORWARD -m physdev --physdev-in tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain -A neutron-linuxbri-INPUT -m physdev --physdev-in tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-linuxbri-ob9543d91-e -A neutron-linuxbri-ib9543d91-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-ib9543d91-e -d 106.0.0.66/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-ib9543d91-e -d 255.255.255.255/32 -p udp -m udp --sport 67 --dport 68 -j RETURN -A neutron-linuxbri-ib9543d91-e -p udp -j RETURN -A neutron-linuxbri-ib9543d91-e -p tcp -j RETURN -A neutron-linuxbri-ib9543d91-e -p icmp -j RETURN -A neutron-linuxbri-ib9543d91-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-ib9543d91-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-ob9543d91-e -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-ob9543d91-e -j neutron-linuxbri-sb9543d91-e -A neutron-linuxbri-ob9543d91-e -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN -A neutron-linuxbri-ob9543d91-e -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP -A neutron-linuxbri-ob9543d91-e -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN -A neutron-linuxbri-ob9543d91-e -j RETURN -A neutron-linuxbri-ob9543d91-e -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP -A neutron-linuxbri-ob9543d91-e -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-linuxbri-sg-fallback -A neutron-linuxbri-sb9543d91-e -s 106.0.0.66/32 -m mac --mac-source FA:16:3E:81:53:80 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-linuxbri-sb9543d91-e -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP -A neutron-linuxbri-sg-chain -m physdev --physdev-out tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-ib9543d91-e -A neutron-linuxbri-sg-chain -m physdev --physdev-in tapb9543d91-e7 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-linuxbri-ob9543d91-e -------------------------------------------------------------------------------------------------------------------------------------------------- #根据接口进行过滤 root@ubuntu:~# iptables -t filter -nL |grep -P "Chain|target|2ae4e683|b9543d91" Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain neutron-filter-top (2 references) target prot opt source destination Chain neutron-linuxbri-FORWARD (1 references) target prot opt source destination neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2ae4e683-e4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2ae4e683-e4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapb9543d91-e7 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-linuxbri-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapb9543d91-e7 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ Chain neutron-linuxbri-INPUT (1 references) target prot opt source destination neutron-linuxbri-o2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2ae4e683-e4 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ neutron-linuxbri-ob9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapb9543d91-e7 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ Chain neutron-linuxbri-OUTPUT (1 references) target prot opt source destination Chain neutron-linuxbri-i2ae4e683-e (1 references) target prot opt source destination Chain neutron-linuxbri-i457326a9-f (1 references) target prot opt source destination Chain neutron-linuxbri-i4ca6dc60-e (1 references) target prot opt source destination Chain neutron-linuxbri-ib9543d91-e (1 references) target prot opt source destination Chain neutron-linuxbri-local (1 references) target prot opt source destination Chain neutron-linuxbri-o2ae4e683-e (2 references) target prot opt source destination neutron-linuxbri-s2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-linuxbri-o457326a9-f (2 references) target prot opt source destination Chain neutron-linuxbri-o4ca6dc60-e (2 references) target prot opt source destination Chain neutron-linuxbri-ob9543d91-e (2 references) target prot opt source destination neutron-linuxbri-sb9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-linuxbri-s2ae4e683-e (1 references) target prot opt source destination Chain neutron-linuxbri-s457326a9-f (1 references) target prot opt source destination Chain neutron-linuxbri-s4ca6dc60-e (1 references) target prot opt source destination Chain neutron-linuxbri-sb9543d91-e (1 references) target prot opt source destination Chain neutron-linuxbri-sg-chain (8 references) target prot opt source destination neutron-linuxbri-i2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2ae4e683-e4 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-o2ae4e683-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2ae4e683-e4 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-ib9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapb9543d91-e7 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-linuxbri-ob9543d91-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapb9543d91-e7 --physdev-is-bridged /* Jump to the VM specific chain. */ Chain neutron-linuxbri-sg-fallback (8 references) target prot opt source destination
Firewall as a Service（FWaaS）；配置enable FWaaS；FWaaS实验(未做) ============================================================================================= Neutron FWaaS 提供虚拟防火墙功能 Firewall as a Service（FWaaS）是 Neutron 的一个高级服务。用户可以用它来创建和管理防火墙，在 subnet 边界上对 layer 3 和 layer 4 的流量进行过滤。 在 Neutron 虚拟 router 上应用防火墙规则，控制进出租户网络的数据。 FWaaS 有三个重要概念：Firewall、Policy 和 Rule。 Firewall 租户能够创建和管理的逻辑防火墙资源。Firewall 必须关联某个 Policy，因此必须先创建 Policy。 Firewall Policy Policy 是 Rule 的集合，Firewall 会按顺序应用 Policy 中的每一条 Rule。 Firewall Rule Rule 是访问控制规则，由源与目的子网 IP、源与目的端口、协议、allow 或 deny 动作组成。例如，我们可以创建一条 Rule，允许外部网络通过 ssh 访问租户网络中的 instance，端口为 22。 启用 FWaaS 因为 FWaaS 是在 router 中实现的，所以 FWaaS 没有单独的 agent。已有的 L3 agent 负责提供所有 FWaaS 功能。 在 Neutron 配置文件 /etc/neutron/neutron.conf 中启用 FWaaS plugin。 root@ubuntu:~# cat /etc/neutron/neutron.conf |grep service_plugins service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPlugin,neutron_vpnaas.services.vpn.plugin.VPNDriverPlugin,neutron_fwaas.services.firewall.fwaas_plugin.FirewallPlugin #service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin #尝试了在neutron.conf开启lbaas plugin和fwaas plugin，neutron-server都起不来，会报错，有可能是没安装相应的模块吧 #在未开启fwaas plugin的前提下， /etc/neutron/fwaas_driver.ini该文件不存在 FWaaS实验 通过 iptables-save 查看 router namespace 的 iptables 规则（需要进入虚拟router的namespace中进行查看） 实验失败，未开启FWaaS功能
#################################################20220103实验中止了，因为环境无法启用FWaaS、LBaaS功能#############################################################################
LBaaS概念（Pool Member、Pool、Virtual IP、 Load Balance Method、Session Persistence、monitor）；配置enable LBaaS；LBaaS 实现机制以及底层网络变化 ============================================================================================= （本节内容缺乏实践，基本都只是推测） Load Balance as a Service（LBaaS）是 Neutron 提供的一项高级网络服务。LBaaS 允许租户在自己的网络中创建和管理 load balancer。 OpenStack Neutron 目前默认通过 HAProxy 软件来实现 LBaaS。 HAProxy 是一个流行的开源 load balancer。 Neutron 也支持其他一些第三方 load balancer。 LBaaS 有三个主要的概念： Pool Member，Pool 和 Virtual IP Pool Member --------------------对应RS Pool Member 是 layer 4 的实体，拥有 IP 地址并通过监听端口对外提供服务。 例如 Pool Member 可以是一个 web server，IP 为 172.16.100.9 并通过 80 端口提供 HTTP 服务。 Pool--------------------------相当于RS集合 Pool 由一组 Pool Member 组成。 这些 Pool Member 通常提供同一类服务。 例如一个 web server pool，包含： web1：172.16.100.9：80 web2：172.16.100.10：80 Virtual IP----------------------对应slb vip Virtual IP 也称作 VIP，是定义在 load balancer 上的 IP 地址。 每个 pool member 都有自己的 IP，但对外服务则是通过 VIP。 load balancer 负责监听外部的连接，并将连接分发到 pool member。 外部 client 只知道 VIP，不知道也不需要关心是否有 pool 或者有多少个 pool member。 ### LBaaS相当于vpc slb；这种slb要对外提供服务，需要和float IP绑定 Load Balance Method 和 Session Persistence 区别：（它们之间的最大区别在于选择 pool member 的阶段不同） 1. Load Balance Method 是为新连接选择 member 的方法 2. Session Persistence 是为同一个 client 的后续连接选择 member 的方法 3. 可选用的算法也不同 LBaaS 可以创建 monitor，用于监控 Pool Member 健康状态。如果某个 member 不能正常工作，monitor 会将其状态设置为 down，从而避免将后续请求转发给它。 相当于slb健康检查 ---------------------------------------------------------------------------------------------------------------------- 配置enable LBaaS Neutron 通过 lbaas plugin 和 lbaas agent 提供 LBaaS 服务。 lbaas plugin 与 Neutron Server 一起运行在控制节点上。lbaas agent 运行在网络节点上。 配置 LBaaS agent 的地方是 /etc/neutron/services/loadbalancer/haproxy/lbaas_agent.ini。 interface_driver 的作用是设置 load balancer 的网络接口驱动 interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver #二选一 配置 LBaaS plugin 在 /etc/neutron/neutron.conf 中设置启用 LBaaS plugin 在 /etc/neutron/neutron_lbaas.conf 中设置 service provider ###除了默认的 HAProxy，Neutron 也支持第三方 provider，比如 radware，VMWareEdge 等。 ---------------------------------------------------------------------------------------------------------------------- LBaaS 实现机制以及底层网络变化 果然，创建了slb，就会创建一个network namespace，并且挂到bridge上 每一个pool，host都会启动一个相应的haproxy进程 haproxy 配置文件保存在 /opt/stack/data/neutron/lbaas/< pool ID>/conf 中 可以看到： 1. frontend 使用的 HTTP 地址为 VIP:80 2. backend 使用的 HTTP 地址为 172.16.100.10:80 和 172.16.100.9:80 3. balance 方法为 roundrobin ####HAProxy 软件
11111111111111111111111111111111111111
