网络错误定位案例 ICMP host *** unreachable - admin prohibited

1. 环境

一台物理服务器 9.115.251.86，上面创建两个虚机，每个虚机两个网卡：

• vm1：eth0 - 9.*.*.232 eth1：10.0.0.14
• vm2: eth0 - 9.8.*.219 eth1:10.0.0.10，上面运行DHCP Agent，管理 dnsmasq，提供 DHCP 服务

两块 eth1 连到物理机上的一个 bridge 上：

bridge name     bridge id               STP enabled     interfaces
mgtbr0          8000.fa8013216b56       no              tap0
                                                        vnet1
                                                        vnet12
                                                        vnet13
                                                        vnet14
                                                        vnet16
                                                        vnet18
                                                        vnet2
                                                        vnet5

2. 错误

vm1 上创建一个虚机 vmchild1，启动时发出 BOOTP 请求去向 DHCP Agent 获取 IP 地址。DHCP Agent 管理的 dnsmasq 可以正常收到 BOOTP 请求，而且发回了请求，但是 vm1 无法收到。

vm2 上 tcpdump：

15:30:14.135874 IP (tos 0x0, ttl 64, id 46594, offset 0, flags [none], proto UDP (17), length 401)
    10.0.0.10.40589 > 10.0.0.14.8472: OTV, flags [I] (0x08), overlay 0, instance 1027
IP (tos 0xc0, ttl 64, id 57625, offset 0, flags [none], proto UDP (17), length 351)
    50-0-0-10.static.sonic.net.bootps > 50-0-0-15.static.sonic.net.bootpc: BOOTP/DHCP, Reply, length 323, xid 0x11f7631f, secs 60, Flags [none]
          Your-IP 50-0-0-15.static.sonic.net
          Server-IP 50-0-0-10.static.sonic.net
          Client-Ethernet-Address fa:16:3e:ba:43:4c (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Offer
            Server-ID Option 54, length 4: 50-0-0-10.static.sonic.net
            Lease-Time Option 51, length 4: 86400
            RN Option 58, length 4: 43200
            RB Option 59, length 4: 75600
            Subnet-Mask Option 1, length 4: 255.255.255.0
            BR Option 28, length 4: 50-0-0-255.static.sonic.net
            Domain-Name Option 15, length 14: "openstacklocal"
            Domain-Name-Server Option 6, length 4: 50-0-0-10.static.sonic.net
            Default-Gateway Option 3, length 4: 50-0-0-1.static.sonic.net
            Classless-Static-Route Option 121, length 13: (60.0.0.0/24:0.0.0.0),(default:50-0-0-1.static.sonic.net)
15:30:14.136118 IP (tos 0xc0, ttl 64, id 10816, offset 0, flags [none], proto ICMP (1), length 429)
    10.0.0.100 > 10.0.0.10: ICMP host 10.0.0.14 unreachable - admin prohibited, length 409
        IP (tos 0x0, ttl 63, id 46594, offset 0, flags [none], proto UDP (17), length 401)
    10.0.0.10.40589 > 10.0.0.14.8472: OTV, flags [I] (0x08), overlay 0, instance 1027

而且发现另外的问题：

root@controller:~/s1# ping 10.0.0.13
PING 10.0.0.13 (10.0.0.13) 56(84) bytes of data.
64 bytes from 10.0.0.13: icmp_seq=1 ttl=64 time=0.630 ms
From 10.0.0.13: icmp_seq=2 Redirect Host(New nexthop: 10.0.0.13)
64 bytes from 10.0.0.13: icmp_seq=2 ttl=64 time=0.628 ms
From 10.0.0.13: icmp_seq=3 Redirect Host(New nexthop: 10.0.0.13)

3.定位

初步推断是某处防火墙阻止了 BOOTP 的包，于是将 vm1，vm2 上的防火墙全部关掉，无效。最后，查看物理机 iptables：

[root@rh65 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:bootps
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:16509
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:16514
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpts:vnc-server:cvsup
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.122.0/24    state RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

发现 FORWARD 阻止了 ICMP 相关的包。

4. 解决

在物理机上运行  service iptables stop 将 iptables 关闭，问题解决。