MPLS Segment Routing – Arista Lab

We have been able to create some nice MPLS labs using GNS3 and Cisco IOS. In my current employer, we use Arista so I wanted to create a lab environment with Arista kit to simulate a MPLS Segment Routing network. Keeping in mind that I try to run everything on my laptop, using GNS3 + Arista is not an option. You need to use the Arista vEOS image in GNS3 and it demands 2GB RAM per device and 1 CPU. In the past, I think I just managed to start two vEOS VMs before my laptop gave up. But Arista offers a version of EOS for containers.

So, what’s the difference between a virtual machine (VM) and a container? Well, searching the internet is going to give you many all answers. In my very simplify way:

  • VM: needs an hypervisor to simulate hardware. It uses kernel and user space. It has a full OS. So it is like simulation a whole server/pc (imagine a standalone house)
  • Container: runs in user space. Set of processes that are isolated from the rest of the system. Containers provide a way to virtualize an OS so that multiple workloads can run on a single OS instance (imagine an apartment in a building)

You just need to register in Arista web page to download a cEOS image.

Regarding MPLS Segment Routing (or SPRING for Juniper) it is an evolution of the standard MPLS, that was originally developed to improve the routing performance in core networks: avoid to make a routing look-ups per packet in core devices was very expensive in 80/90s (my very simplify way). MPLS started to being deployed around end 90s and became a defacto technology in all service providers. More info here.

Segment Routing is still based in labels, but adds improvements as it doesnt need a protocol for label exchange (one less thing to worry about). As well, it is based in “source routing” as the sources chooses the path and encodes it in the packet.

There are many sources in the internet that can explain MPLS SR better than me like all these:

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/seg_routing/configuration/xe-3s/segrt-xe-3s-book/intro-seg-routing.html

https://www.segment-routing.net/tutorials/

As we are going to use Arista, I based my learning in these presentations:

https://ripe77.ripe.net/presentations/16-20181015-SegmentRouting.pdf

https://www.netnod.se/sites/default/files/2018-03/Peter%20Lundqvist_Arista_8.pdf

And reading more Arista docs.

All the code and how to build the lab is here:

https://github.com/thomarite/ceos-testing

So what we need and what we are going to use in this lab:

  • IPv4 (yeah, I should start working in IPv6…)
  • IGP: we use ISIS
  • Label Distribution: ISIS-SR
  • BGP: using loobacks as best practices and using IGP for building a full-mesh
  • L3/2VPN: EVPN
  • All devices are PE

So let’s build the basic IP connectivity for r01:

!
hostname r01
!
interface Ethernet1
no switchport
ip address 10.0.10.1/30
!
interface Ethernet2
no switchport
ip address 10.0.12.1/30
!
interface Loopback1
description CORE Loopback
ip address 10.0.0.1/32
!
ip routing
!

Now let’s build our IGP with ISIS. We are going to use our Lo1 IP as network ID for each router. As well, we will keep it simple and define all routers as ISIS L2. We dont need anything fancy. We just want ISIS to build our iBGP peering. We will enable ISIS in the core interfaces (in this simple lab, all links and loopbacks)

!
router isis CORE
net 49.0000.0001.0010.0000.0000.0001.00  <-- BASED IN Lo1 !!!
is-type level-2
log-adjacency-changes
set-overload-bit on-startup wait-for-bgp timeout 180
!
interface Ethernet1
no switchport
ip address 10.0.10.1/30
isis enable CORE
isis metric 40
isis network point-to-point
!
interface Ethernet2
no switchport
ip address 10.0.12.1/30
isis enable CORE
isis metric 50
isis network point-to-point
!
interface Loopback1
description CORE Loopback
ip address 10.0.0.1/32
isis enable CORE
isis metric 1
!

It is seems there is a bug in the cEOS I am using as “show isis neighbors” fails but the routing is actually correct. Let’s see from r22:

r22#show ip route
VRF: default
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked
Gateway of last resort is not set
I L2 10.0.0.1/32 [115/131] via 10.0.10.9, Ethernet1
I L2 10.0.0.2/32 [115/91] via 10.0.10.9, Ethernet1
I L2 10.0.0.3/32 [115/91] via 10.0.23.1, Ethernet2
I L2 10.0.0.4/32 [115/51] via 10.0.23.1, Ethernet2
I L2 10.0.0.5/32 [115/41] via 10.0.10.9, Ethernet1
C 10.0.0.6/32 is directly connected, Loopback1
I L2 10.0.10.0/30 [115/130] via 10.0.10.9, Ethernet1
I L2 10.0.10.4/30 [115/90] via 10.0.23.1, Ethernet2
C 10.0.10.8/30 is directly connected, Ethernet1
I L2 10.0.12.0/30 [115/140] via 10.0.23.1, Ethernet2
I L2 10.0.13.0/30 [115/90] via 10.0.10.9, Ethernet1
C 10.0.23.0/30 is directly connected, Ethernet2
r22#
r22# show logging
...
Log Buffer:
May 24 16:18:22 r22 SuperServer: %SYS-5-SYSTEM_RESTARTED: System restarted
May 24 16:24:29 r22 ConfigAgent: %SYS-5-CONFIG_E: Enter configuration mode from console by root on vty4 (UnknownIpAddr)
May 24 16:24:29 r22 ConfigAgent: %SYS-5-CONFIG_I: Configured from console by root on vty4 (UnknownIpAddr)
May 24 16:24:29 r22 ConfigAgent: %SYS-5-CONFIG_STARTUP: Startup config saved from system:/running-config by root on vty4 (UnknownIpAddr).
May 24 16:24:39 r22 Isis: %ISIS-4-ISIS_ADJCHG: L2 Neighbor State Change for SystemID 0000.0000.0004 on eth2 to UP
May 24 16:24:42 r22 Isis: %ISIS-4-ISIS_ADJCHG: L2 Neighbor State Change for SystemID 0000.0000.0005 on eth1 to UP
May 24 16:26:34 r22 ConfigAgent: %SYS-5-CONFIG_STARTUP: Startup config saved from system:/running-config by root on vty4 (UnknownIpAddr).
r22#
r22#show isis neighbors
% Internal error
% To see the details of this error, run the command 'show error 2'

Let’s build BGP, from r01 is like this:

!
router bgp 100
router-id 10.0.0.1
graceful-restart restart-time 300
graceful-restart
maximum-paths 2
neighbor AS100-CORE peer group
neighbor AS100-CORE remote-as 100
neighbor AS100-CORE next-hop-self
neighbor AS100-CORE update-source Loopback1
neighbor AS100-CORE timers 2 6
neighbor AS100-CORE additional-paths receive
neighbor AS100-CORE additional-paths send any
neighbor AS100-CORE password 7 Nmg+xbfVkywN7BBIllK5yw==
neighbor AS100-CORE send-community standard extended
neighbor AS100-CORE maximum-routes 0
neighbor 10.0.0.2 peer group AS100-CORE
neighbor 10.0.0.2 description R02
neighbor 10.0.0.3 peer group AS100-CORE
neighbor 10.0.0.3 description R11
neighbor 10.0.0.4 peer group AS100-CORE
neighbor 10.0.0.4 description R12
neighbor 10.0.0.5 peer group AS100-CORE
neighbor 10.0.0.5 description R21
neighbor 10.0.0.6 peer group AS100-CORE
neighbor 10.0.0.6 description R22
!

So once we have configured BGP in all routers, we should see a full mesh between all routers. This is from r22:

r22#show ip bgp summary
BGP summary information for VRF default
Router identifier 10.0.0.6, local AS number 100
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
R01 10.0.0.1 4 100 7 7 0 0 00:00:05 Estab 0 0
R02 10.0.0.2 4 100 7 7 0 0 00:00:05 Estab 0 0
R11 10.0.0.3 4 100 7 7 0 0 00:00:05 Estab 0 0
R12 10.0.0.4 4 100 6 7 0 0 00:00:04 Estab 0 0
R21 10.0.0.5 4 100 6 7 0 0 00:00:04 Estab 0 0
r22#

Now, enable MPLS and SR extension in ISIS:

!
mpls ip
!
mpls label range isis-sr 800000 65536
!
router isis CORE
  segment-routing mpls
    router-id 10.0.0.1  <-- based on Lo1 in each router
    no shutdown
!
interface Loopback1
  description CORE Loopback
  node-segment ipv4 index 1  <-- this has to be different in each node!!!
!

And you should see 5 ISIS-SR tunnels from each router. From r22:

r22#show isis segment-routing tunnel
Index Endpoint Nexthop Interface Labels TI-LFA
tunnel index

1 10.0.0.2/32 10.0.10.9 Ethernet1 [ 800002 ] -
2 10.0.0.3/32 10.0.23.1 Ethernet2 [ 800003 ] -
3 10.0.0.4/32 10.0.23.1 Ethernet2 [ 3 ] -
4 10.0.0.5/32 10.0.10.9 Ethernet1 [ 3 ] -
5 10.0.0.1/32 10.0.10.9 Ethernet1 [ 800001 ] -
r22#

As you can see above, the labels are based on the base index (800000) defined in the “mpls label range” command and the “node-segment index” defined in the loopback interface. So the label that identifies uniquely r01 is 800000 + 1 = 800001. The label “3” means you are a Penultime-Hop-P router and you remove the label to save a label look-up in the egress router.

Now, let’s configure EVPN for L2/L3VPN deployment in our MPLS network. From r01 should be:

!
service routing protocols model multi-agent --> you will have to reboot
!
router bgp 100
!
address-family evpn
neighbor default encapsulation mpls next-hop-self source-interface Loopback1
neighbor 10.0.0.2 activate
neighbor 10.0.0.3 activate
neighbor 10.0.0.4 activate
neighbor 10.0.0.5 activate
neighbor 10.0.0.6 activate
!

So once this is configured in all routers, we should see again a full mesh of EVPN BGP peers. From r12 this time:

r12#show bgp evpn summary
BGP summary information for VRF default
Router identifier 10.0.0.4, local AS number 100
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
R01 10.0.0.1 4 100 1254 1251 0 0 00:03:27 Estab 1 1
R02 10.0.0.2 4 100 1111 1107 0 0 00:03:27 Estab 1 1
R11 10.0.0.3 4 100 961 962 0 0 00:03:27 Estab 1 1
R21 10.0.0.5 4 100 884 888 0 0 00:03:27 Estab 1 1
R22 10.0.0.6 4 100 814 811 0 0 00:03:27 Estab 1 1
r12#

Now, let’s create a L3VPN with CUST-A vrf. We define it in all routers. For r01 should be:

!
vrf instance CUST-A
rd 100:1
!
interface Loopback2
vrf CUST-A
ip address 192.168.0.1/32   <-- each device has a unique one
!
ip routing vrf CUST-A
!
router bgp 100
!
vrf CUST-A
rd 100:1
route-target import evpn 100:1
route-target export evpn 100:1
network 192.168.0.1/32

Let’s see if the routing works from r12

r12#
r12#show bgp evpn
BGP routing table information for VRF default
Router identifier 10.0.0.4, local AS number 100
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path 
RD: 100:1 ip-prefix 192.168.0.1/32 10.0.0.1 - 100 0 i 
RD: 100:1 ip-prefix 192.168.0.2/32 10.0.0.2 - 100 0 i 
RD: 100:1 ip-prefix 192.168.0.3/32 10.0.0.3 - 100 0 i 
RD: 100:1 ip-prefix 192.168.0.5/32 10.0.0.5 - 100 0 i 
RD: 100:1 ip-prefix 192.168.0.6/32 10.0.0.6 - 100 0 i
r12#
r12#show ip route vrf CUST-A
VRF: CUST-A
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked
Gateway of last resort is not set
B I 192.168.0.1/32 [200/0] via 10.0.0.1/32, IS-IS SR tunnel index 5, label 116384
via 10.0.10.5, Ethernet1, label 800001
B I 192.168.0.2/32 [200/0] via 10.0.0.2/32, IS-IS SR tunnel index 2, label 116384
via 10.0.10.5, Ethernet1, label 800002
B I 192.168.0.3/32 [200/0] via 10.0.0.3/32, IS-IS SR tunnel index 3, label 100000
via 10.0.10.5, Ethernet1, label imp-null(3)
C 192.168.0.4/32 is directly connected, Loopback2
B I 192.168.0.5/32 [200/0] via 10.0.0.5/32, IS-IS SR tunnel index 4, label 116384
via 10.0.23.2, Ethernet2, label 800005
B I 192.168.0.6/32 [200/0] via 10.0.0.6/32, IS-IS SR tunnel index 1, label 116384
via 10.0.23.2, Ethernet2, label imp-null(3)
r12#

So, all looks good. EVPN table shows all the prefixes for rd 100:1 and the routing table for CUST-A shows all Lo2 defined in each router.

BTW, I am not able to ping inside the VRF, I think it is something related to the broadcast of ARP:

UPDATE: Arista confirms that cEOS-lab doesn’t support MPLS dataplane. I need to use vEOS (vagrant). So that means I dont think my laptop has enough resources to build this lab in vEOS 🙁

r01#ping vrf CUST-A ip 192.168.0.6 interface loopback 2
PING 192.168.0.6 (192.168.0.6) from 192.168.0.1 lo2: 72(100) bytes of data.
--- 192.168.0.6 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 40ms
r01#

-- from other session in r01 --

r01#bash
bash-4.2# ip netns exec ns-CUST-A tcpdump -i lo2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo2, link-type EN10MB (Ethernet), capture size 262144 bytes
^C12:46:03.324918 02:00:00:00:00:00 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.0.6 tell 192.168.0.1, length 28
12:46:04.348750 02:00:00:00:00:00 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.0.6 tell 192.168.0.1, length 28
12:46:05.376723 02:00:00:00:00:00 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.0.6 tell 192.168.0.1, length 28
3 packets captured
3 packets received by filter
0 packets dropped by kernel
bash-4.2#

-- from other session in r22, we dont see anything --

r22#bash
bash-4.2# ip netns exec ns-CUST-A tcpdump -i lo2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo2, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
bash-4.2#