DEV Community

rhira777
rhira777

Posted on

[SONiC Virtual Switch] EVPN-VXLANv4 / BGP unnumbered

Summary

I tried BGP unnumbered and the combination with EVPN-VXLAN in the SONiC on CML environment that we had previously built.

In this case, the virtual switch version of SONiC.202205 was configured with a minimum of communication checks in mind, and were able to achieve IPv4 forwarding using BGP unnumbered and L2 forwarding using EVPN-VXLAN.

The following is a record of that.

The main information that I was referring to is as follows:

[Edgecore SONiC] BGP Unnumbered
https://support.edge-core.com/hc/en-us/articles/900002377366--Edgecore-SONiC-BGP-Unnumbered

[Edgecore SONiC] EVPN L2 VxLAN
https://support.edge-core.com/hc/en-us/articles/900002720523--Edgecore-SONiC-EVPN-L2-VxLAN

[Edgecore SONiC] VLAN & Inter-VLAN Routing
https://support.edge-core.com/hc/en-us/articles/900000210426-VLAN-Inter-VLAN-Routing

SONiC USER MANUAL
https://github.com/sonic-net/SONiC/blob/master/doc/SONiC-User-Manual.md

SONiC Command Line Interface Guide
https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md

Vxlan SONiC
https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md

Detail of what I did

Network diagram(Topology)

Image description

Network information

node hostname Loopback0 ASN
sonicvsvm-0 R001 10.0.0.1/32 65001
sonicvsvm-1 R002 10.0.0.2/32 65001
sonicvsvm-2 R011 10.0.0.11/32 65000
sonicvsvm-3 R012 10.0.0.12/32 65000
sonicvsvm-4 R013 10.0.0.13/32 65000
sonicvsvm-5 R014 10.0.0.14/32 65000
sonicvsvm-6 R101 10.0.0.101/32 65001
sonicvsvm-7 R102 10.0.0.102/32 65001

node hostname VLAN VNI
sonicvsvm-6 R101 2 5002
sonicvsvm-7 R102 2 5002
node interface MAC address VLAN IP address VRF
csr1000v-0 Gi2 0000.0000.0101 untag 192.168.2.101 101-2
Gi3 0000.0000.0102 untag 192.168.2.102 102-2

SONiC OS configuration (Excerpts)

  • save and reload (execute as appropriate)
node command
sonicvsvm-0~7 sudo config save -y
sudo config reload -fy

  • transit link: IPv6 link-local
node command
sonicvsvm-0~5 sudo config ipv6 enable link-local
sonicvsvm-6/7 sudo config interface ipv6 enable use-link-local-only Ethernet0
sudo config interface ipv6 enable use-link-local-only Ethernet4
  • loopback: IPv4
node command
sonicvsvm-0 sudo config interface ip add Loopback0 10.0.0.1/32
sonicvsvm-1 sudo config interface ip add Loopback0 10.0.0.2/32
sonicvsvm-2 sudo config interface ip add Loopback0 10.0.0.11/32
sonicvsvm-3 sudo config interface ip add Loopback0 10.0.0.12/32
sonicvsvm-4 sudo config interface ip add Loopback0 10.0.0.13/32
sonicvsvm-5 sudo config interface ip add Loopback0 10.0.0.14/32
sonicvsvm-6 sudo config interface ip add Loopback0 10.0.0.101/32
sudo config interface ip add Loopback0 10.255.0.101/32 secondary
sonicvsvm-7 sudo config interface ip add Loopback0 10.0.0.102/32
sudo config interface ip add Loopback0 10.255.0.102/32 secondary

  • VLAN
node command
sonicvsvm-6/7 sudo config vlan add 2
sudo config vlan member add -u 2 Ethernet8
  • VXLAN (EVPN compatible)
node command
sonicvsvm-6 sudo config vxlan add VXLAN 10.255.0.101
sonicvsvm-7 sudo config vxlan add VXLAN 10.255.0.102
sonicvsvm-6/7 sudo config vxlan evpn_nvo add NVO VXLAN
sudo config vxlan map add VXLAN 2 5002

FRRouting configuration (Excerpts)

  • sonicvsvm-0/1
no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 neighbor Ethernet12 interface peer-group TRANSPORT
 bgp listen range 10.0.0.0/8 peer-group SERVICE
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  neighbor SERVICE route-reflector-client
 exit-address-family
exit

Enter fullscreen mode Exit fullscreen mode
  • sonicvsvm-2/3/4/5
no router bgp 65100
router bgp 65000
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
exit

Enter fullscreen mode Exit fullscreen mode
  • sonicvsvm-6/7
no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor 10.0.0.1 peer-group SERVICE
 neighbor 10.0.0.2 peer-group SERVICE
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  advertise-all-vni
 exit-address-family
exit

Enter fullscreen mode Exit fullscreen mode

Confirmation of operation

  • csr1000v-0
HOST#ping vrf 101-2 192.168.2.102 source 192.168.2.101 repeat 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 192.168.2.102, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.101
.!!
Success rate is 66 percent (2/3), round-trip min/avg/max = 5/5/6 ms
HOST#
HOST#!#ping:ok
Enter fullscreen mode Exit fullscreen mode

  • csr1000v-0 ~ sonicvsvm-7
    Image description

  • sonicvsvm-5 ~ sonicvsvm-7
    Image description

Supplemental information and others

Status of IPv4 FIB

In the preparation process, IPv4 routes exchanged by BGP were being imported, but IPv4 communication for those routes was failing. In this case, the entries displayed in "show ip route" were not displayed in "ip route show".
This problem was resolved by reloading once.

  • before reloading
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>r 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:05:53
B>r 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:05:53
B>r 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
C>* 10.0.0.101/32 is directly connected, Loopback0, 10:02:07
B>r 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19

admin@R101:~$
admin@R101:~$ ip route show
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms

admin@R101:~$
Enter fullscreen mode Exit fullscreen mode
  • after reloading
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
B>* 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25
B>* 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:17
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:17
C>* 10.0.0.101/32 is directly connected, Loopback0, 00:06:34
B>* 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25

admin@R101:~$
admin@R101:~$ ip route show
10.0.0.1 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.2 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.11 nhid 227 via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 proto bgp src 10.0.0.101 metric 20
10.0.0.12 nhid 228 via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 proto bgp src 10.0.0.101 metric 20
10.0.0.13 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.14 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.102 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.
64 bytes from 10.0.0.102: icmp_seq=1 ttl=61 time=3.81 ms
64 bytes from 10.0.0.102: icmp_seq=2 ttl=61 time=3.29 ms
64 bytes from 10.0.0.102: icmp_seq=3 ttl=61 time=3.05 ms

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.045/3.381/3.806/0.316 ms
admin@R101:~$
Enter fullscreen mode Exit fullscreen mode

Status of bridge table

After successful L2 communication, the EVPN Route Type 2 routes exchanged by BGP were imported, but their entries were not displayed in output such as "show mac".
In this environment, it is possible that this is a display problem or that unicast communication also success as BUM forwarding. It may be possible to proceed to isolate this point, for example, by adding NVEs.

  • sonicvsvm-6
admin@R101:~$ #ping:ok
admin@R101:~$
admin@R101:~$ show mac
  No.    Vlan  MacAddress         Port       Type
-----  ------  -----------------  ---------  -------
    1       2  00:00:00:00:01:01  Ethernet8  Dynamic
Total number of entries 1
admin@R101:~$
admin@R101:~$ vtysh

Hello, this is FRRouting (version 8.2.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

R101#
R101# show bgp l2vpn evpn
BGP table version is 9, local router ID is 10.0.0.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.101:1
*> [2]:[0]:[48]:[00:00:00:00:01:01]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
*> [3]:[0]:[32]:[10.255.0.101]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
Route Distinguisher: 10.0.0.102:1
* i[2]:[0]:[48]:[00:00:00:00:01:02]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
* i[3]:[0]:[32]:[10.255.0.102]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8

Displayed 4 out of 6 total prefixes
R101#
Enter fullscreen mode Exit fullscreen mode

Top comments (1)

Collapse
 
bluesky007 profile image
Arthur • Edited

Hi, rhira777. This is really amazing and helps me a lot! And I have a question that how does R101 forward data packet to 192.168.2.101 when ping from 192.168.2.102? I mean through what way R101 knows mac ip and port binding. Thanks for the help!