Linux+MPLS-Part1

In November 2020, I got an email from the FRR email list about using MPLS with FRR. And the answer that you could do already natively (and easily) MPLS in Linux dumbfound me. So I add in my to-do list, try MPLS in Linux as per the blog. So all credits to the author, that’s a great job.

So reading the blog, I learned that the kernel supported MPLS since 4.3 (I am using 5.10) and creating VRF support was challenging until Cumulus did it. Thanks! So since April 2017 there is full support for L3VPNs in Linux… I’m getting a bit late in the wagon.

Anyway, I want to test myself and see if I can make it work. I downloaded the repo from the author to start working on it.

So I am following the same steps as him and will start with a lab consisting of static LSP. This is the diagram:

Main differences in my lab are:

1- I use libvirt instead of VirtualBox

2- I am using debian10 buster64 as VM

This affect the Vagrant file and the script to configure the static LSP. The libvirt_ commands I am using in Vagrantfile are ignored as I am not able to name the interfaces as I want. As well, I had to change the IP addressing as I had collisions with .1. And debian/buster64 has specific interfaces names that I have to use.

So, now we can turn up the lab.

/mpls-linux/lab1-static-lsps$ vagrant up
 Bringing machine 'r1' up with 'libvirt' provider…
 Bringing machine 'r2' up with 'libvirt' provider…
 Bringing machine 'r3' up with 'libvirt' provider…
 ==> r2: Checking if box 'debian/buster64' version '10.4.0' is up to date…
 ==> r3: Checking if box 'debian/buster64' version '10.4.0' is up to date…
 ==> r1: Checking if box 'debian/buster64' version '10.4.0' is up to date…
 ==> r1: Creating image (snapshot of base box volume).
 ==> r2: Creating image (snapshot of base box volume).
 ==> r3: Creating image (snapshot of base box volume).
 ==> r2: Creating domain with the following settings…
 ==> r1: Creating domain with the following settings…
...
/mpls-linux/lab1-static-lsps master$ vagrant status
 Current machine states:
 r1                        running (libvirt)
 r2                        running (libvirt)
 r3                        running (libvirt)

So we can check R1. One important detail here, is how we can defined a static route to reach R3 loopback and it is encapsulated in MPLS with label 100.

/mpls-linux/lab1-static-lsps$ vagrant ssh r1
...
vagrant@R1:~$ lsmod | grep mpls
 mpls_iptunnel          16384  1
 mpls_router            36864  1 mpls_iptunnel
 ip_tunnel              24576  1 mpls_router
 vagrant@R1:~$ 
 vagrant@R1:~$ ip route
 default via 192.168.121.1 dev ens5 proto dhcp src 192.168.121.124 metric 1024 
 172.20.15.3  encap mpls  100 via 192.168.12.102 dev ens6 
 192.168.12.0/24 dev ens6 proto kernel scope link src 192.168.12.101 
 192.168.121.0/24 dev ens5 proto kernel scope link src 192.168.121.124 
 192.168.121.1 dev ens5 proto dhcp scope link src 192.168.121.124 metric 1024 
 vagrant@R1:~$ 
 vagrant@R1:~$ ip -4 a
 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet 172.20.15.1/32 scope global lo
        valid_lft forever preferred_lft forever
 2: ens5:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.121.124/24 brd 192.168.121.255 scope global dynamic ens5
        valid_lft 3204sec preferred_lft 3204sec
 3: ens6:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.12.101/24 brd 192.168.12.255 scope global ens6
        valid_lft forever preferred_lft forever
 vagrant@R1:~$ 

Now check R2 as it is our P router between R1 and R3 as per diagram. Important bit here is “ip -M route show”. This shows the MPLS routing label that is based in labels. In the standard “ip route” you dont seen any reference to MPLS.

vagrant@R2:~$ ip -4 a
 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet 172.20.15.2/32 scope global lo
        valid_lft forever preferred_lft forever
 2: ens5:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.121.103/24 brd 192.168.121.255 scope global dynamic ens5
        valid_lft 2413sec preferred_lft 2413sec
 3: ens6:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.12.102/24 brd 192.168.12.255 scope global ens6
        valid_lft forever preferred_lft forever
 4: ens7:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.23.102/24 brd 192.168.23.255 scope global ens7
        valid_lft forever preferred_lft forever
 vagrant@R2:~$ ip route
 default via 192.168.121.1 dev ens5 proto dhcp src 192.168.121.103 metric 1024 
 192.168.12.0/24 dev ens6 proto kernel scope link src 192.168.12.102 
 192.168.23.0/24 dev ens7 proto kernel scope link src 192.168.23.102 
 192.168.121.0/24 dev ens5 proto kernel scope link src 192.168.121.103 
 192.168.121.1 dev ens5 proto dhcp scope link src 192.168.121.103 metric 1024 
 vagrant@R2:~$ 
 vagrant@R2:~$ lsmod | grep mpls
 mpls_router            36864  0
 ip_tunnel              24576  1 mpls_router
 vagrant@R2:~$ 
 vagrant@R2:~$ ip -M route show
 100 via inet 192.168.23.101 dev ens7 
 200 via inet 192.168.12.101 dev ens6 
 vagrant@R2:~$ 

So let’s see if pinging the loopback in R1 and R3 gets labelled traffic:

R1 to R3 (on R2)

root@R2:/home/vagrant# tcpdump -i ens6 -U -w - | tee mpls-r1tor3.pcap | tcpdump -r -
 reading from file -, link-type EN10MB (Ethernet)
 tcpdump: listening on ens6, link-type EN10MB (Ethernet), capture size 262144 bytes
 17:14:01.284942 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:03.300756 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:05.284915 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:07.183328 MPLS (label 100, exp 0, [S], ttl 64) IP 192.168.12.101 > 172.20.15.3: ICMP echo request, id 1771, seq 1, length 64
 17:14:07.300556 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:08.186983 MPLS (label 100, exp 0, [S], ttl 64) IP 192.168.12.101 > 172.20.15.3: ICMP echo request, id 1771, seq 2, length 64
 17:14:09.188867 MPLS (label 100, exp 0, [S], ttl 64) IP 192.168.12.101 > 172.20.15.3: ICMP echo request, id 1771, seq 3, length 64

I can see the labelled packet from R1 to R2 with label 100 as expected, but I dont see any “echo reply”…..

But ping is successful based on R1:

vagrant@R1:~$ ping 172.20.15.3
 PING 172.20.15.3 (172.20.15.3) 56(84) bytes of data.
 64 bytes from 172.20.15.3: icmp_seq=1 ttl=63 time=0.746 ms
 64 bytes from 172.20.15.3: icmp_seq=2 ttl=63 time=1.18 ms
 64 bytes from 172.20.15.3: icmp_seq=3 ttl=63 time=1.11 ms
 64 bytes from 172.20.15.3: icmp_seq=4 ttl=63 time=0.728 ms

Something is wrong. As per pic below, with tcpdump in all interfaces, R3 is seeing the echo request from a different source (not R1).

And if I ping using R1 loopback, I can’t see anything leaving R1 ens6 interface.

vagrant@R1:~$ ping 172.20.15.3 -I lo         
 PING 172.20.15.3 (172.20.15.3) from 172.20.15.1 lo: 56(84) bytes of data.
 ^C
 --- 172.20.15.3 ping statistics ---
 25 packets transmitted, 0 received, 100% packet loss, time 576ms

Based on the original blog post, this should work. The main difference here is I am using libvirt. Need to carry on investigating

This is my IP config, 23.1 is my laptop:

9: virbr3:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr3
        valid_lft forever preferred_lft forever
 10: virbr8:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.12.1/24 brd 192.168.12.255 scope global virbr8
        valid_lft forever preferred_lft forever
 11: virbr9:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.23.1/24 brd 192.168.23.255 scope global virbr9
        valid_lft forever preferred_lft forever

NOTES:

How to scp files from vagrant box: link

$ vagrant plugin install vagrant-scp
$ vagrant scp r2:~/*.pcap .

How to ssh to a vagrant box without using “vagran ssh”: link

# save the config to a file 
vagrant ssh-config > vagrant-ssh 

# run ssh with the file
ssh -F vagrant-ssh default

# update your .gitignore for not tracking this file!!!!

How to write and read tcpdump at the same time:

# tcpdump -i ens7 -U -w - | tee mpls-r3tor1.pcap | tcpdump -r -

UPDATE:

Ok, I have tried again. I rebooted my laptop, rebuilt the VMs, etc. And now it works

9: virbr3:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr3
        valid_lft forever preferred_lft forever
 10: virbr8:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.12.1/24 brd 192.168.12.255 scope global virbr8
        valid_lft forever preferred_lft forever
 11: virbr9:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.23.1/24 brd 192.168.23.255 scope global virbr9
        valid_lft forever preferred_lft forever
 root@athens:/boot# uname -a
 Linux athens 5.9.0-5-amd64 #1 SMP Debian 5.9.15-1 (2020-12-17) x86_64 GNU/Linux
 root@athens:/boot# 

I can see now clearly, how the ICMP request packet is encapsulated with MPLS tag 100 from R1 to R2 (ens6 interface), then the label is popped in R2, and you can see the same ICMP request leaving R2 via ens7 to R3.

Then the ICMP reply is encapsulated with MPLS tag 200 in R3 to R2 (ens7) and again, the labels is popped in R2, and you see the packet again from R2 (ens6) to R1.

So this test is successful at the end although not sure what I have been doing wrong before.