17 January 2021

In November 2020, I got an email from the FRR email list about using MPLS with FRR. And the answer that you could do already natively (and easily) MPLS in Linux dumbfound me. So I add in my to-do list, try MPLS in Linux as per the blog. So all credits to the author, that’s a great job.

So reading the blog, I learned that the kernel supported MPLS since 4.3 (I am using 5.10) and creating VRF support was challenging until Cumulus did it. Thanks! So since April 2017 there is full support for L3VPNs in Linux… I’m getting a bit late in the wagon.

Anyway, I want to test myself and see if I can make it work. I downloaded the repo from the author to start working on it.

So I am following the same steps as him and will start with a lab consisting of static LSP. This is the diagram:

Main differences in my lab are:

1- I use libvirt instead of VirtualBox

2- I am using debian10 buster64 as VM

This affect the Vagrant file and the script to configure the static LSP. The libvirt_ commands I am using in Vagrantfile are ignored as I am not able to name the interfaces as I want. As well, I had to change the IP addressing as I had collisions with .1. And debian/buster64 has specific interfaces names that I have to use.

So, now we can turn up the lab.

/mpls-linux/lab1-static-lsps$ vagrant up
 Bringing machine 'r1' up with 'libvirt' provider…
 Bringing machine 'r2' up with 'libvirt' provider…
 Bringing machine 'r3' up with 'libvirt' provider…
 ==> r2: Checking if box 'debian/buster64' version '10.4.0' is up to date…
 ==> r3: Checking if box 'debian/buster64' version '10.4.0' is up to date…
 ==> r1: Checking if box 'debian/buster64' version '10.4.0' is up to date…
 ==> r1: Creating image (snapshot of base box volume).
 ==> r2: Creating image (snapshot of base box volume).
 ==> r3: Creating image (snapshot of base box volume).
 ==> r2: Creating domain with the following settings…
 ==> r1: Creating domain with the following settings…
...
/mpls-linux/lab1-static-lsps master$ vagrant status
 Current machine states:
 r1                        running (libvirt)
 r2                        running (libvirt)
 r3                        running (libvirt)

So we can check R1. One important detail here, is how we can defined a static route to reach R3 loopback and it is encapsulated in MPLS with label 100.

/mpls-linux/lab1-static-lsps$ vagrant ssh r1
...
vagrant@R1:~$ lsmod | grep mpls
 mpls_iptunnel          16384  1
 mpls_router            36864  1 mpls_iptunnel
 ip_tunnel              24576  1 mpls_router
 vagrant@R1:~$ 
 vagrant@R1:~$ ip route
 default via 192.168.121.1 dev ens5 proto dhcp src 192.168.121.124 metric 1024 
 172.20.15.3  encap mpls  100 via 192.168.12.102 dev ens6 
 192.168.12.0/24 dev ens6 proto kernel scope link src 192.168.12.101 
 192.168.121.0/24 dev ens5 proto kernel scope link src 192.168.121.124 
 192.168.121.1 dev ens5 proto dhcp scope link src 192.168.121.124 metric 1024 
 vagrant@R1:~$ 
 vagrant@R1:~$ ip -4 a
 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet 172.20.15.1/32 scope global lo
        valid_lft forever preferred_lft forever
 2: ens5:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.121.124/24 brd 192.168.121.255 scope global dynamic ens5
        valid_lft 3204sec preferred_lft 3204sec
 3: ens6:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.12.101/24 brd 192.168.12.255 scope global ens6
        valid_lft forever preferred_lft forever
 vagrant@R1:~$

Now check R2 as it is our P router between R1 and R3 as per diagram. Important bit here is “ip -M route show”. This shows the MPLS routing label that is based in labels. In the standard “ip route” you dont seen any reference to MPLS.

vagrant@R2:~$ ip -4 a
 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet 172.20.15.2/32 scope global lo
        valid_lft forever preferred_lft forever
 2: ens5:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.121.103/24 brd 192.168.121.255 scope global dynamic ens5
        valid_lft 2413sec preferred_lft 2413sec
 3: ens6:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.12.102/24 brd 192.168.12.255 scope global ens6
        valid_lft forever preferred_lft forever
 4: ens7:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.23.102/24 brd 192.168.23.255 scope global ens7
        valid_lft forever preferred_lft forever
 vagrant@R2:~$ ip route
 default via 192.168.121.1 dev ens5 proto dhcp src 192.168.121.103 metric 1024 
 192.168.12.0/24 dev ens6 proto kernel scope link src 192.168.12.102 
 192.168.23.0/24 dev ens7 proto kernel scope link src 192.168.23.102 
 192.168.121.0/24 dev ens5 proto kernel scope link src 192.168.121.103 
 192.168.121.1 dev ens5 proto dhcp scope link src 192.168.121.103 metric 1024 
 vagrant@R2:~$ 
 vagrant@R2:~$ lsmod | grep mpls
 mpls_router            36864  0
 ip_tunnel              24576  1 mpls_router
 vagrant@R2:~$ 
 vagrant@R2:~$ ip -M route show
 100 via inet 192.168.23.101 dev ens7 
 200 via inet 192.168.12.101 dev ens6 
 vagrant@R2:~$

So let’s see if pinging the loopback in R1 and R3 gets labelled traffic:

R1 to R3 (on R2)

root@R2:/home/vagrant# tcpdump -i ens6 -U -w - | tee mpls-r1tor3.pcap | tcpdump -r -
 reading from file -, link-type EN10MB (Ethernet)
 tcpdump: listening on ens6, link-type EN10MB (Ethernet), capture size 262144 bytes
 17:14:01.284942 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:03.300756 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:05.284915 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:07.183328 MPLS (label 100, exp 0, [S], ttl 64) IP 192.168.12.101 > 172.20.15.3: ICMP echo request, id 1771, seq 1, length 64
 17:14:07.300556 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:61:f2.8001, length 35
 17:14:08.186983 MPLS (label 100, exp 0, [S], ttl 64) IP 192.168.12.101 > 172.20.15.3: ICMP echo request, id 1771, seq 2, length 64
 17:14:09.188867 MPLS (label 100, exp 0, [S], ttl 64) IP 192.168.12.101 > 172.20.15.3: ICMP echo request, id 1771, seq 3, length 64

I can see the labelled packet from R1 to R2 with label 100 as expected, but I dont see any “echo reply”…..

But ping is successful based on R1:

vagrant@R1:~$ ping 172.20.15.3
 PING 172.20.15.3 (172.20.15.3) 56(84) bytes of data.
 64 bytes from 172.20.15.3: icmp_seq=1 ttl=63 time=0.746 ms
 64 bytes from 172.20.15.3: icmp_seq=2 ttl=63 time=1.18 ms
 64 bytes from 172.20.15.3: icmp_seq=3 ttl=63 time=1.11 ms
 64 bytes from 172.20.15.3: icmp_seq=4 ttl=63 time=0.728 ms

Something is wrong. As per pic below, with tcpdump in all interfaces, R3 is seeing the echo request from a different source (not R1).

And if I ping using R1 loopback, I can’t see anything leaving R1 ens6 interface.

vagrant@R1:~$ ping 172.20.15.3 -I lo         
 PING 172.20.15.3 (172.20.15.3) from 172.20.15.1 lo: 56(84) bytes of data.
 ^C
 --- 172.20.15.3 ping statistics ---
 25 packets transmitted, 0 received, 100% packet loss, time 576ms

Based on the original blog post, this should work. The main difference here is I am using libvirt. Need to carry on investigating

This is my IP config, 23.1 is my laptop:

9: virbr3:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr3
        valid_lft forever preferred_lft forever
 10: virbr8:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.12.1/24 brd 192.168.12.255 scope global virbr8
        valid_lft forever preferred_lft forever
 11: virbr9:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.23.1/24 brd 192.168.23.255 scope global virbr9
        valid_lft forever preferred_lft forever

NOTES:

How to scp files from vagrant box: link

$ vagrant plugin install vagrant-scp
$ vagrant scp r2:~/*.pcap .

How to ssh to a vagrant box without using “vagran ssh”: link

# save the config to a file 
vagrant ssh-config > vagrant-ssh 

# run ssh with the file
ssh -F vagrant-ssh default

# update your .gitignore for not tracking this file!!!!

How to write and read tcpdump at the same time:

# tcpdump -i ens7 -U -w - | tee mpls-r3tor1.pcap | tcpdump -r -

UPDATE:

Ok, I have tried again. I rebooted my laptop, rebuilt the VMs, etc. And now it works

9: virbr3:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr3
        valid_lft forever preferred_lft forever
 10: virbr8:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.12.1/24 brd 192.168.12.255 scope global virbr8
        valid_lft forever preferred_lft forever
 11: virbr9:  mtu 1500 qdisc noqueue state UP group default qlen 1000
     inet 192.168.23.1/24 brd 192.168.23.255 scope global virbr9
        valid_lft forever preferred_lft forever
 root@athens:/boot# uname -a
 Linux athens 5.9.0-5-amd64 #1 SMP Debian 5.9.15-1 (2020-12-17) x86_64 GNU/Linux
 root@athens:/boot#

I can see now clearly, how the ICMP request packet is encapsulated with MPLS tag 100 from R1 to R2 (ens6 interface), then the label is popped in R2, and you can see the same ICMP request leaving R2 via ens7 to R3.

Then the ICMP reply is encapsulated with MPLS tag 200 in R3 to R2 (ens7) and again, the labels is popped in R2, and you see the packet again from R2 (ens6) to R1.

So this test is successful at the end although not sure what I have been doing wrong before.

It is never too late. I have never tried “Roscon de Reyes” before. But this year (although over a week later) I wanted to make it as it is the only thing I miss from (my child time) Christmas (Sorry Santa, you dont belong to my culture)

So I used as inspiration this video.

Ingredients for dough:

580g Strong white flour (I guess you can use plain too)
180 ml milk (I use semi-skim)
100g sugar
2 egg
7g instant yeast (1 sacket – I can’t get fresh yeast)
80g butter (room temperature)
zest of 1 orange
zest of 1 lemon
pinch of salt
1 tsp of vanilla paste (I dont have “agua de azahar”)

Decoration:

Cherry glacier
Italian mixed peel (orange and lemon glacier peel)
almond flakes
Egg wash

Ingredients for the filling / cream:

500ml of double cream (cold from the fridge)
50g sugar
20ml milk

Process:

In a bowl, put the flour, sugar, yest, zest of lemon, zest of orange, pinch of salt, vanilla paste. Mix everything.

Make a whole in the middle, add the milk, eggs. Mix everything until nearly combine. Finally the butter. Mix again, you will reach a point it is not possible in the bowl.

Put the dough in a work surface. Start knelling a bit. Once your hands are quite sticky with the dough. Stop for a couple of minutes while you scrap the dough from your hands into the main ball. Start knelling again, it should be much easier now and you should get a soft dough after 5-7 minutes.

Put a couple of oil drops in the bowl and spread it so we are going to put the dough in it. Cover with plastic film and let it rest until double in size. It can be over a couple of hours as it is winter.

Once the dough is double in size, remove it from the bowl. Remove the air bubbles knelling a bit, let it rest of a couple of minutes in the bowl.

Take the dough and make a whole in the middle. Spin the dough so you create a “wheel”. Put it in a try with baking paper. Let it rest until double in size again (this time will be quicker)

Now, start preparing the filling. Put in a bowl the double cream, sugar and milk. I used a hand blender and it was perfectly fine. The cream became solid quite quickly to be honest. Give it a taste, you can add more sugar if you want. Let it rest in the fridge.

Once the ring-dough has double up. Pre-heat the oven at 180C. Egg wash the dough very well, this will make it golden. Then add as much Italian mixed peel, glacier cherries and almond flakes as you can. The egg should help to stick it.

Once the oven is hot. Put the “roscon” in until golden. My case was nearly 30 minutes.

Once it is golden, remove from the oven. Let it cool down. With a bread knife, cut it in two slices. Be careful dont break the top, put it aside. Now spread all your cream / “nata” in the base. Put back the top over the cream. And that’s it.

It brought me very good memories (and it is tasty!!!)

Day: 17 January 2021

Linux+MPLS-Part1

Roscon-de-Reyes