vxlan + 多个vrf
一开始只有一个vrf
[root@evpn2 ~]# vtysh Hello, this is FRRouting (version 7.3-MyOwnFRRVersion). Copyright 1996-2005 Kunihiro Ishiguro, et al. evpn2.novalocal# show running config % Unknown command: show running config evpn2.novalocal# show running-config Building configuration... Current configuration: ! frr version 7.3-MyOwnFRRVersion frr defaults traditional hostname evpn2.novalocal log file /var/log/frr/bgpd.log ! vrf evpn-vrf vni 100 exit-vrf ! router bgp 9999 bgp router-id 10.10.18.212 bgp bestpath as-path multipath-relax neighbor fabric peer-group neighbor fabric remote-as external neighbor 10.10.18.209 peer-group fabric neighbor 10.10.18.209 update-source 10.10.18.212 ! address-family l2vpn evpn neighbor fabric activate advertise-all-vni exit-address-family ! router bgp 9999 vrf evpn-vrf ! address-family ipv4 unicast network 0.0.0.0/0 network 9.9.9.0/24 exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast exit-address-family ! line vty ! end
现在再加一个
#添加vni 1000，作为l3vni sudo ip link add br1000 type bridge sudo ip link add vxlan1000 type vxlan id 1000 local 10.10.18.212 dstport 4789 nolearning sudo ip link set br1000 up sudo ip link set vxlan1000 up sudo ip link set vxlan1000 master br1000 sudo ip link set dev br1000 address 00:00:01:02:03:06 //不和以前的冲突 ip link add evpn-vrf2 type vrf table 1000 ip link set evpn-vrf2 up ip link set br1000 master evpn-vrf2
evpn2.novalocal# show running-config
Building configuration...
Current configuration:
!
frr version 7.3-MyOwnFRRVersion
frr defaults traditional
hostname evpn2.novalocal
log file /var/log/frr/bgpd.log
!
vrf evpn-vrf
vni 100
exit-vrf
!
vrf evpn-vrf2
vni 1000
exit-vrf
!
router bgp 9999
bgp router-id 10.10.18.212
bgp bestpath as-path multipath-relax
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor 10.10.18.209 peer-group fabric
neighbor 10.10.18.209 update-source 10.10.18.212
!
address-family l2vpn evpn
neighbor fabric activate
advertise-all-vni
exit-address-family
!
router bgp 9999 vrf evpn-vrf
!
address-family ipv4 unicast
network 0.0.0.0/0
network 9.9.9.0/24
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
exit-address-family
!
router bgp 9999 vrf evpn-vrf2
!
address-family ipv4 unicast
network 0.0.0.0/0
exit-address-family
!
line vty
!
end
[root@evpn2 ~]# ip route show vrf evpn-vrf2 什么都没有 ip link add vrf2-in type veth peer name vrf2-out ip link set vrf2-in up ip link set vrf2-out up ip link set vrf2-in master evpn-vrf2 ip addr add 5.5.5.251/24 dev vrf2-in ip addr add 5.5.5.252/24 dev vrf2-out ip route add default via 5.5.5.252 dev vrf2-in table 1000 [root@evpn2 ~]# ip route show vrf evpn-vrf2 default via 5.5.5.252 dev vrf2-in 5.5.5.0/24 dev vrf2-in proto kernel scope link src 5.5.5.251 [root@evpn2 ~]#
[root@evpn2 ~]# ping 5.5.5.252 PING 5.5.5.252 (5.5.5.252) 56(84) bytes of data. 64 bytes from 5.5.5.252: icmp_seq=1 ttl=64 time=0.051 ms 64 bytes from 5.5.5.252: icmp_seq=2 ttl=64 time=0.055 ms ^C --- 5.5.5.252 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 31ms rtt min/avg/max/mdev = 0.051/0.053/0.055/0.002 ms [root@evpn2 ~]# ip vrf exec evpn-vrf2 ping 5.5.5.2521 ^C [root@evpn2 ~]# ip vrf exec evpn-vrf2 ping 5.5.5.251 PING 5.5.5.251 (5.5.5.251) 56(84) bytes of data. 64 bytes from 5.5.5.251: icmp_seq=1 ttl=64 time=0.054 ms 64 bytes from 5.5.5.251: icmp_seq=2 ttl=64 time=0.056 ms ^C --- 5.5.5.251 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 2ms rtt min/avg/max/mdev = 0.054/0.055/0.056/0.001 ms [root@evpn2 ~]#
在另一个vtep2上查看
evpn1# show evpn vni VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF 20 L2 vxlan20 1 4 0 evpn-vrf 10 L2 vxlan10 1 4 1 evpn-vrf 100 L3 vxlan100 1 1 n/a evpn-vrf evpn1# show bgp evpn route BGP table version is 10, local router ID is 10.10.18.209 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Extended Community Route Distinguisher: 9.9.9.254:3 *> [5]:[0]:[0]:[0.0.0.0] 10.10.18.212 0 0 9999 i RT:9999:100 ET:8 Rmac:00:00:01:02:03:05 -------------------------100是vni的id *> [5]:[0]:[24]:[9.9.9.0] 10.10.18.212 0 0 9999 i RT:9999:100 ET:8 Rmac:00:00:01:02:03:05 Route Distinguisher: 10.10.18.209:2 *> [3]:[0]:[32]:[10.10.18.209] 10.10.18.209 32768 i ET:8 RT:8888:10 Route Distinguisher: 10.10.18.209:4 *> [3]:[0]:[32]:[10.10.18.209] 10.10.18.209 32768 i ET:8 RT:8888:20 Route Distinguisher: 10.10.18.212:2 *> [3]:[0]:[32]:[10.10.18.212] 10.10.18.212 0 9999 i RT:9999:10 ET:8 Displayed 5 prefixes (5 paths) evpn1# show ip route vrf evpn-vrf Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued route, r - rejected route VRF evpn-vrf: B>* 0.0.0.0/0 [20/0] via 10.10.18.212, br100 onlink, 04w4d20h C>* 2.2.2.0/24 is directly connected, br10, 05w2d01h C>* 3.3.3.0/24 is directly connected, br20, 05w1d00h B>* 9.9.9.0/24 [20/0] via 10.10.18.212, br100 onlink, 05w0d05h evpn1# exit
没看1000 vni
配置vtep
evpn1# show running-config Building configuration... Current configuration: ! frr version 7.3-MyOwnFRRVersion frr defaults traditional hostname evpn2.novalocal log file /var/log/frr/bgpd.log hostname evpn1 ! vrf evpn-vrf vni 100 exit-vrf ! router bgp 8888 bgp router-id 10.10.18.209 bgp bestpath as-path multipath-relax neighbor fabric peer-group neighbor fabric remote-as external neighbor 10.10.18.212 peer-group fabric neighbor 10.10.18.212 update-source 10.10.18.209 ! address-family l2vpn evpn neighbor fabric activate advertise-all-vni exit-address-family ! router bgp 8888 vrf evpn-vrf ! address-family l2vpn evpn advertise ipv4 unicast exit-address-family ! line vty ! end evpn1# conf t evpn1(config)# router bgp 8888 evpn1(config-router)# router bgp 8888 vrf evpn-vrf2 evpn1(config-router)# exit evpn1(config)# vrf evpn-vrf2 evpn1(config-vrf)# vni 1000 evpn1(config-vrf)# exit evpn1(config)# wr m % Unknown command: wr m evpn1(config)# exit evpn1# wr Note: this version of vtysh never writes vtysh.conf Building Configuration... Configuration saved to /etc/frr/zebra.conf Configuration saved to /etc/frr/ospfd.conf Configuration saved to /etc/frr/bgpd.conf Configuration saved to /etc/frr/pimd.conf Configuration saved to /etc/frr/fabricd.conf Configuration saved to /etc/frr/staticd.conf evpn1# exit
evpn1# show running-config
Building configuration...
Current configuration:
!
frr version 7.3-MyOwnFRRVersion
frr defaults traditional
hostname evpn2.novalocal
log file /var/log/frr/bgpd.log
hostname evpn1
!
vrf evpn-vrf
vni 100
exit-vrf
!
vrf evpn-vrf2
vni 1000
exit-vrf
!
router bgp 8888
bgp router-id 10.10.18.209
bgp bestpath as-path multipath-relax
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor 10.10.18.212 peer-group fabric
neighbor 10.10.18.212 update-source 10.10.18.209
!
address-family l2vpn evpn
neighbor fabric activate
advertise-all-vni
exit-address-family
!
router bgp 8888 vrf evpn-vrf
!
address-family l2vpn evpn
advertise ipv4 unicast
exit-address-family
!
router bgp 8888 vrf evpn-vrf2
!
line vty
!
end
evpn1#
Hello, this is FRRouting (version 7.3-MyOwnFRRVersion). Copyright 1996-2005 Kunihiro Ishiguro, et al. evpn1# show ip route vrf evpn-vrf Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued route, r - rejected route VRF evpn-vrf: B>* 0.0.0.0/0 [20/0] via 10.10.18.212, br100 onlink, 04w4d21h //没有br1000 C>* 2.2.2.0/24 is directly connected, br10, 05w2d01h C>* 3.3.3.0/24 is directly connected, br20, 05w1d01h B>* 9.9.9.0/24 [20/0] via 10.10.18.212, br100 onlink, 05w0d05h evpn1#
[root@evpn1 ~]# ip link add vxlan88 type vxlan id 88 local 10.10.18.209 dstport 4789 nolearning [root@evpn1 ~]# ip link set vxlan88 up [root@evpn1 ~]# ip link set vxlan88 master br30 [root@evpn1 ~]# ip link set br30 master evpn-vrf2 evpn1# show evpn vni VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF 20 L2 vxlan20 1 4 0 evpn-vrf 88 L2 vxlan88 2 4 0 default 10 L2 vxlan10 2 4 1 evpn-vrf 100 L3 vxlan100 1 1 n/a evpn-vrf 1000 L3 vxlan1000 0 0 n/a evpn-vrf2 evpn1# exit [root@evpn1 ~]# systemctl restart frr ------------------从起frr哦，在bgp通告之前要先设置vxlan的master [root@evpn1 ~]# vtysh Hello, this is FRRouting (version 7.3-MyOwnFRRVersion). Copyright 1996-2005 Kunihiro Ishiguro, et al. evpn1# show evpn vni VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF 20 L2 vxlan20 1 4 0 evpn-vrf 88 L2 vxlan88 2 4 0 evpn-vrf2 10 L2 vxlan10 2 4 1 evpn-vrf 100 L3 vxlan100 1 1 n/a evpn-vrf 1000 L3 vxlan1000 0 0 n/a evpn-vrf2 evpn1# show ip route vrf evpn-vrf2 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued route, r - rejected route VRF evpn-vrf2: C>* 192.168.3.0/24 is directly connected, br30, 00:00:36
[root@evpn1 ~]# ip link set dev br1000 address 00:00:01:02:03:08 --------------不能和vtep上的一样哦 [root@evpn1 ~]# ip route show vrf evpn-vrf2 192.168.3.0/24 dev br30 proto kernel scope link src 192.168.3.254 -------------------路由通告没有收到
[root@evpn1 ~]# ip netns exec host3 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ----------------还是无法通哦
8
查看vtep1
vtep2上 [root@evpn1 ~]# ip route show vrf evpn-vrf2 default via 10.10.18.212 dev br1000 proto bgp metric 20 onlink ----------------收到路由通告 192.168.3.0/24 dev br30 proto kernel scope link src 192.168.3.254 [root@evpn1 ~]#
[root@evpn1 ~]# ip netns exec host3 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ---------------------还是无法ping通
vtep1上
[root@evpn1 ~]# tcpdump -i vxlan1000 -eennvv
tcpdump: listening on vxlan1000, link-type EN10MB (Ethernet), capture size 262144 bytes
16:30:47.719967 00:00:01:02:03:08 > 00:00:01:02:03:06, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 41974, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.3.3 > 8.8.8.8: ICMP echo request, id 5211, seq 312, length 64
16:30:48.759998 00:00:01:02:03:08 > 00:00:01:02:03:06, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 41990, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.3.3 > 8.8.8.8: ICMP echo request, id 5211, seq 313, length 64
16:30:49.799995 00:00:01:02:03:08 > 00:00:01:02:03:06, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 42069, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.3.3 > 8.8.8.8: ICMP echo request, id 5211, seq 314, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
[root@evpn1 ~]#
[root@evpn2 ~]# ip vrf exec evpn-vrf2 tcpdump -i vrf2-in -eennvv tcpdump: listening on vrf2-in, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel -------------没有抓到包，说明数据没有转给vf2-in [root@evpn2 ~]#
evpn2.novalocal# show bgp evpn route BGP table version is 1, local router ID is 10.10.18.212 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Extended Community Route Distinguisher: 5.5.5.251:4 *> [5]:[0]:[0]:[0.0.0.0] 10.10.18.212 0 32768 i ET:8 RT:9999:1000 Rmac:00:00:01:02:03:06 Route Distinguisher: 9.9.9.254:3 *> [5]:[0]:[0]:[0.0.0.0] 10.10.18.212 0 32768 i ET:8 RT:9999:100 Rmac:00:00:01:02:03:05 *> [5]:[0]:[24]:[9.9.9.0] 10.10.18.212 0 32768 i ET:8 RT:9999:100 Rmac:00:00:01:02:03:05 Route Distinguisher: 10.10.18.209:4 *> [3]:[0]:[32]:[10.10.18.209] 10.10.18.209 0 8888 i RT:8888:10 ET:8 Route Distinguisher: 10.10.18.209:5 *> [2]:[0]:[48]:[b6:7a:bc:9e:4e:95] 10.10.18.209 0 8888 i RT:8888:20 RT:8888:100 ET:8 Rmac:00:00:01:02:03:07 *> [2]:[0]:[48]:[b6:7a:bc:9e:4e:95]:[32]:[3.3.3.2] 10.10.18.209 0 8888 i RT:8888:20 RT:8888:100 ET:8 Rmac:00:00:01:02:03:07 *> [2]:[0]:[48]:[b6:7a:bc:9e:4e:95]:[128]:[fe80::b47a:bcff:fe9e:4e95] 10.10.18.209 0 8888 i RT:8888:20 ET:8 ----------------------------------同一个vtep上不同vrf的rt et相同 *> [3]:[0]:[32]:[10.10.18.209] 10.10.18.209 0 8888 i RT:8888:20 ET:8 Route Distinguisher: 10.10.18.209:6 *> [3]:[0]:[32]:[10.10.18.209] 10.10.18.209 0 8888 i RT:8888:88 ET:8----------------------------------------同一个vetep上不同vrf的rt et相同
Route Distinguisher: 10.10.18.212:2 *>
[3]:[0]:[32]:[10.10.18.212]
10.10.18.212 32768 i
ET:8 RT:9999:10 Displayed 10 prefixes (10 paths) evpn2.novalocal#
[root@evpn1 ~]# ip netns exec host3 ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
evpn2.novalocal# show bgp evpn route BGP table version is 1, local router ID is 10.10.18.212 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Extended Community Route Distinguisher: 5.5.5.251:3 *> [5]:[0]:[0]:[0.0.0.0] 10.10.18.212 0 32768 i ET:8 RT:9999:1000 Rmac:00:00:01:02:03:06 ----------------------- 1000是vpi的id Route Distinguisher: 9.9.9.254:2 *> [5]:[0]:[0]:[0.0.0.0] 10.10.18.212 0 32768 i ET:8 RT:9999:100 Rmac:00:00:01:02:03:05 *> [5]:[0]:[24]:[9.9.9.0] 10.10.18.212 0 32768 i ET:8 RT:9999:100 Rmac:00:00:01:02:03:05
vtep1上查看路由
[root@evpn2 ~]# ip vrf exec evpn-vrf2 tcpdump -i vrf2-in -eennvv tcpdump: listening on vrf2-in, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel
也抓不到包
[root@evpn2 ~]# sysctl -p /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@evpn2 ~]#
[root@evpn2 ~]# ip a add 6.6.6.254/24 dev vrf2-out
[root@evpn2 ~]# ip a add 6.6.6.253/24 dev vrf2-in
[root@evpn2 ~]# ip r show vrf evpn-vrf2
default via 6.6.6.254 dev vrf2-in
6.6.6.0/24 dev vrf2-in proto kernel scope link src 6.6.6.253
更改地址后还是不行
[root@evpn2 ~]# ip rule show 0: from all lookup local 1000: from all lookup [l3mdev-table] 32766: from all lookup main 32767: from all lookup default 默认的表是1000 ip link add evpn-vrf2 type vrf table 1000
重新构造vrf 把
ip link add evpn-vrf2 type vrf table 1000
换成
ip link add evpn-vrf2 type vrf table 20
nft add table nat nft add chain nat prerouting { type nat hook prerouting priority 0 \; } nft add chain nat postrouting { type nat hook postrouting priority 100 \; } nft add rule nat postrouting oifname default_g1 counter masquerade nft add rule nat postrouting oifname enp1s0 counter masquerade [root@evpn2 ~]# nft add table nat2 [root@evpn2 ~]# [root@evpn2 ~]# nft add chain nat2 prerouting { type nat hook prerouting priority 0 \; } [root@evpn2 ~]# nft add chain nat2 postrouting { type nat hook postrouting priority 100 \; } [root@evpn2 ~]# nft add rule nat2 postrouting oifname vrf-in counter masquerade
