Summary
I tried BGP unnumbered and the combination with EVPN-VXLAN in the SONiC on CML environment that we had previously built.
In this case, the virtual switch version of SONiC.202205 was configured with a minimum of communication checks in mind, and were able to achieve IPv4 forwarding using BGP unnumbered and L2 forwarding using EVPN-VXLAN.
The following is a record of that.
The main information that I was referring to is as follows:
[Edgecore SONiC] BGP Unnumbered
https://support.edge-core.com/hc/en-us/articles/900002377366--Edgecore-SONiC-BGP-Unnumbered
[Edgecore SONiC] EVPN L2 VxLAN
https://support.edge-core.com/hc/en-us/articles/900002720523--Edgecore-SONiC-EVPN-L2-VxLAN
[Edgecore SONiC] VLAN & Inter-VLAN Routing
https://support.edge-core.com/hc/en-us/articles/900000210426-VLAN-Inter-VLAN-Routing
SONiC USER MANUAL
https://github.com/sonic-net/SONiC/blob/master/doc/SONiC-User-Manual.md
SONiC Command Line Interface Guide
https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md
Vxlan SONiC
https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md
Detail of what I did
Network diagram(Topology)
Network information
node | hostname | Loopback0 | ASN |
---|---|---|---|
sonicvsvm-0 | R001 | 10.0.0.1/32 | 65001 |
sonicvsvm-1 | R002 | 10.0.0.2/32 | 65001 |
sonicvsvm-2 | R011 | 10.0.0.11/32 | 65000 |
sonicvsvm-3 | R012 | 10.0.0.12/32 | 65000 |
sonicvsvm-4 | R013 | 10.0.0.13/32 | 65000 |
sonicvsvm-5 | R014 | 10.0.0.14/32 | 65000 |
sonicvsvm-6 | R101 | 10.0.0.101/32 | 65001 |
sonicvsvm-7 | R102 | 10.0.0.102/32 | 65001 |
node | hostname | VLAN | VNI |
---|---|---|---|
sonicvsvm-6 | R101 | 2 | 5002 |
sonicvsvm-7 | R102 | 2 | 5002 |
node | interface | MAC address | VLAN | IP address | VRF |
---|---|---|---|---|---|
csr1000v-0 | Gi2 | 0000.0000.0101 | untag | 192.168.2.101 | 101-2 |
Gi3 | 0000.0000.0102 | untag | 192.168.2.102 | 102-2 |
SONiC OS configuration (Excerpts)
- save and reload (execute as appropriate)
node | command |
---|---|
sonicvsvm-0~7 | sudo config save -y |
sudo config reload -fy |
- transit link: IPv6 link-local
node | command |
---|---|
sonicvsvm-0~5 | sudo config ipv6 enable link-local |
sonicvsvm-6/7 | sudo config interface ipv6 enable use-link-local-only Ethernet0 |
sudo config interface ipv6 enable use-link-local-only Ethernet4 |
- loopback: IPv4
node | command |
---|---|
sonicvsvm-0 | sudo config interface ip add Loopback0 10.0.0.1/32 |
sonicvsvm-1 | sudo config interface ip add Loopback0 10.0.0.2/32 |
sonicvsvm-2 | sudo config interface ip add Loopback0 10.0.0.11/32 |
sonicvsvm-3 | sudo config interface ip add Loopback0 10.0.0.12/32 |
sonicvsvm-4 | sudo config interface ip add Loopback0 10.0.0.13/32 |
sonicvsvm-5 | sudo config interface ip add Loopback0 10.0.0.14/32 |
sonicvsvm-6 | sudo config interface ip add Loopback0 10.0.0.101/32 |
sudo config interface ip add Loopback0 10.255.0.101/32 secondary | |
sonicvsvm-7 | sudo config interface ip add Loopback0 10.0.0.102/32 |
sudo config interface ip add Loopback0 10.255.0.102/32 secondary |
- VLAN
node | command |
---|---|
sonicvsvm-6/7 | sudo config vlan add 2 |
sudo config vlan member add -u 2 Ethernet8 |
- VXLAN (EVPN compatible)
node | command |
---|---|
sonicvsvm-6 | sudo config vxlan add VXLAN 10.255.0.101 |
sonicvsvm-7 | sudo config vxlan add VXLAN 10.255.0.102 |
sonicvsvm-6/7 | sudo config vxlan evpn_nvo add NVO VXLAN |
sudo config vxlan map add VXLAN 2 5002 |
FRRouting configuration (Excerpts)
- sonicvsvm-0/1
no router bgp 65100
router bgp 65001
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
neighbor SERVICE peer-group
neighbor SERVICE remote-as internal
neighbor SERVICE update-source Loopback0
neighbor TRANSPORT peer-group
neighbor TRANSPORT remote-as external
neighbor TRANSPORT capability extended-nexthop
neighbor Ethernet0 interface peer-group TRANSPORT
neighbor Ethernet4 interface peer-group TRANSPORT
neighbor Ethernet8 interface peer-group TRANSPORT
neighbor Ethernet12 interface peer-group TRANSPORT
bgp listen range 10.0.0.0/8 peer-group SERVICE
address-family ipv4 unicast
redistribute connected
neighbor TRANSPORT activate
neighbor TRANSPORT allowas-in
exit-address-family
address-family l2vpn evpn
neighbor SERVICE activate
neighbor SERVICE route-reflector-client
exit-address-family
exit
- sonicvsvm-2/3/4/5
no router bgp 65100
router bgp 65000
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
neighbor TRANSPORT peer-group
neighbor TRANSPORT remote-as external
neighbor TRANSPORT capability extended-nexthop
neighbor Ethernet0 interface peer-group TRANSPORT
neighbor Ethernet4 interface peer-group TRANSPORT
neighbor Ethernet8 interface peer-group TRANSPORT
address-family ipv4 unicast
redistribute connected
neighbor TRANSPORT activate
neighbor TRANSPORT allowas-in
exit-address-family
exit
- sonicvsvm-6/7
no router bgp 65100
router bgp 65001
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
neighbor SERVICE peer-group
neighbor SERVICE remote-as internal
neighbor SERVICE update-source Loopback0
neighbor TRANSPORT peer-group
neighbor TRANSPORT remote-as external
neighbor TRANSPORT capability extended-nexthop
neighbor 10.0.0.1 peer-group SERVICE
neighbor 10.0.0.2 peer-group SERVICE
neighbor Ethernet0 interface peer-group TRANSPORT
neighbor Ethernet4 interface peer-group TRANSPORT
address-family ipv4 unicast
redistribute connected
neighbor TRANSPORT activate
neighbor TRANSPORT allowas-in
exit-address-family
address-family l2vpn evpn
neighbor SERVICE activate
advertise-all-vni
exit-address-family
exit
Confirmation of operation
- csr1000v-0
HOST#ping vrf 101-2 192.168.2.102 source 192.168.2.101 repeat 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 192.168.2.102, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.101
.!!
Success rate is 66 percent (2/3), round-trip min/avg/max = 5/5/6 ms
HOST#
HOST#!#ping:ok
Supplemental information and others
Status of IPv4 FIB
In the preparation process, IPv4 routes exchanged by BGP were being imported, but IPv4 communication for those routes was failing. In this case, the entries displayed in "show ip route" were not displayed in "ip route show".
This problem was resolved by reloading once.
- before reloading
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
B>r 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
r via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
r via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:05:53
B>r 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:05:53
B>r 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
r via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
r via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
C>* 10.0.0.101/32 is directly connected, Loopback0, 10:02:07
B>r 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
r via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
admin@R101:~$
admin@R101:~$ ip route show
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.
--- 10.0.0.102 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms
admin@R101:~$
- after reloading
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
B>* 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
* via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
* via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
B>* 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
* via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25
B>* 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:17
* via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:17
C>* 10.0.0.101/32 is directly connected, Loopback0, 00:06:34
B>* 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
* via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25
admin@R101:~$
admin@R101:~$ ip route show
10.0.0.1 nhid 226 proto bgp src 10.0.0.101 metric 20
nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.2 nhid 226 proto bgp src 10.0.0.101 metric 20
nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.11 nhid 227 via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 proto bgp src 10.0.0.101 metric 20
10.0.0.12 nhid 228 via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 proto bgp src 10.0.0.101 metric 20
10.0.0.13 nhid 226 proto bgp src 10.0.0.101 metric 20
nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.14 nhid 226 proto bgp src 10.0.0.101 metric 20
nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.102 nhid 226 proto bgp src 10.0.0.101 metric 20
nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.
64 bytes from 10.0.0.102: icmp_seq=1 ttl=61 time=3.81 ms
64 bytes from 10.0.0.102: icmp_seq=2 ttl=61 time=3.29 ms
64 bytes from 10.0.0.102: icmp_seq=3 ttl=61 time=3.05 ms
--- 10.0.0.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.045/3.381/3.806/0.316 ms
admin@R101:~$
Status of bridge table
After successful L2 communication, the EVPN Route Type 2 routes exchanged by BGP were imported, but their entries were not displayed in output such as "show mac".
In this environment, it is possible that this is a display problem or that unicast communication also success as BUM forwarding. It may be possible to proceed to isolate this point, for example, by adding NVEs.
- sonicvsvm-6
admin@R101:~$ #ping:ok
admin@R101:~$
admin@R101:~$ show mac
No. Vlan MacAddress Port Type
----- ------ ----------------- --------- -------
1 2 00:00:00:00:01:01 Ethernet8 Dynamic
Total number of entries 1
admin@R101:~$
admin@R101:~$ vtysh
Hello, this is FRRouting (version 8.2.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
R101#
R101# show bgp l2vpn evpn
BGP table version is 9, local router ID is 10.0.0.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.101:1
*> [2]:[0]:[48]:[00:00:00:00:01:01]
10.255.0.101 32768 i
ET:8 RT:65001:5002
*> [3]:[0]:[32]:[10.255.0.101]
10.255.0.101 32768 i
ET:8 RT:65001:5002
Route Distinguisher: 10.0.0.102:1
* i[2]:[0]:[48]:[00:00:00:00:01:02]
10.255.0.102 0 100 0 i
RT:65001:5002 ET:8
*>i 10.255.0.102 0 100 0 i
RT:65001:5002 ET:8
* i[3]:[0]:[32]:[10.255.0.102]
10.255.0.102 0 100 0 i
RT:65001:5002 ET:8
*>i 10.255.0.102 0 100 0 i
RT:65001:5002 ET:8
Displayed 4 out of 6 total prefixes
R101#
Top comments (1)
Hi, rhira777. This is really amazing and helps me a lot! And I have a question that how does R101 forward data packet to 192.168.2.101 when ping from 192.168.2.102? I mean through what way R101 knows mac ip and port binding. Thanks for the help!