T.I.L – Page 39 – Today I Learned

Linux+MPLS-Part4

Finally I am trying to setup MPLS L3VPN.

Again, I am following the author post but adapting it to my environment using libvirt instead of VirtualBox and Debian10 as VM. All my data is here.

This is the diagram for the lab:

Difference from lab3 and lab2. We have P1, that is a pure P router, only handling labels, it doesnt do any BGP.

This time all devices FRR config are generated automatically via gen_frr_config.py (in lab2 all config was manual).

Again the environment is configured via Vagrant file + l3vpn_provisioning script. This is mix of lab2 (install FRR), lab3 (define VRFs) and lab1 (configure MPLS at linux level).

So after some tuning, everything is installed, routing looks correct (although I dont know why but I have to reload FRR to get the proper generated BGP config in PE1 and PE2. P1 is fine).

So let’s see PE1:

IGP (IS-IS) is up:

PE1# show isis neighbor 
 Area ISIS:
   System Id           Interface   L  State        Holdtime SNPA
   P1                  ens8        2  Up            30       2020.2020.2020
 PE1# 
 PE1# exit
 root@PE1:/home/vagrant#

BGP is up to PE2 and we can see routes received in AF IPv4VPN:

PE1# 
 PE1# show bgp summary 
 IPv4 Unicast Summary:
 BGP router identifier 172.20.5.1, local AS number 65010 vrf-id 0
 BGP table version 0
 RIB entries 0, using 0 bytes of memory
 Peers 1, using 21 KiB of memory
 Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
 172.20.5.2      4      65010       111       105        0    0    0 01:39:14            0        0
 Total number of neighbors 1
 IPv4 VPN Summary:
 BGP router identifier 172.20.5.1, local AS number 65010 vrf-id 0
 BGP table version 0
 RIB entries 11, using 2112 bytes of memory
 Peers 1, using 21 KiB of memory
 Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
 172.20.5.2      4      65010       111       105        0    0    0 01:39:14            2        2
 Total number of neighbors 1
 PE1#

Check routing tables, we can see prefixes in both VRFs, so that’s good. And the labels needed.

PE1# show ip route vrf all 
 Codes: K - kernel route, C - connected, S - static, R - RIP,
        O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
        T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
        F - PBR, f - OpenFabric,
        > - selected route, * - FIB route, q - queued, r - rejected, b - backup
 VRF default:
 C>* 172.20.5.1/32 is directly connected, lo, 02:19:16
 I>* 172.20.5.2/32 [115/30] via 192.168.66.102, ens8, label 17, weight 1, 02:16:10
 I>* 172.20.5.5/32 [115/20] via 192.168.66.102, ens8, label implicit-null, weight 1, 02:18:34
 I   192.168.66.0/24 [115/20] via 192.168.66.102, ens8 inactive, weight 1, 02:18:34
 C>* 192.168.66.0/24 is directly connected, ens8, 02:19:16
 I>* 192.168.77.0/24 [115/20] via 192.168.66.102, ens8, label implicit-null, weight 1, 02:18:34
 C>* 192.168.121.0/24 is directly connected, ens5, 02:19:16
 K>* 192.168.121.1/32 [0/1024] is directly connected, ens5, 02:19:16
 VRF vrf_cust1:
 C>* 192.168.11.0/24 is directly connected, ens6, 02:19:05
 B>  192.168.23.0/24 [200/0] via 172.20.5.2 (vrf default) (recursive), label 80, weight 1, 02:13:32
 via 192.168.66.102, ens8 (vrf default), label 17/80, weight 1, 02:13:32 
 VRF vrf_cust2:
 C>* 192.168.12.0/24 is directly connected, ens7, 02:19:05
 B>  192.168.24.0/24 [200/0] via 172.20.5.2 (vrf default) (recursive), label 81, weight 1, 02:13:32
 via 192.168.66.102, ens8 (vrf default), label 17/81, weight 1, 02:13:32
 PE1#

Now check LDP and MPLS labels. Everything looks sane. We have LDP labels for P1 (17) and PE2 (18). And labels for each VFR.

PE1# show mpls table 
  Inbound Label  Type  Nexthop         Outbound Label  
 
 16             LDP   192.168.66.102  implicit-null   
  17             LDP   192.168.66.102  implicit-null   
  18             LDP   192.168.66.102  17              
  80             BGP   vrf_cust1       -               
  81             BGP   vrf_cust2       -               
 PE1# 
 PE1# show mpls ldp neighbor 
 AF   ID              State       Remote Address    Uptime
 ipv4 172.20.5.5      OPERATIONAL 172.20.5.5      02:20:20
 PE1# 
 PE1# 
 PE1# show mpls ldp binding  
 AF   Destination          Nexthop         Local Label Remote Label  In Use
 ipv4 172.20.5.1/32        172.20.5.5      imp-null    16                no
 ipv4 172.20.5.2/32        172.20.5.5      18          17               yes
 ipv4 172.20.5.5/32        172.20.5.5      16          imp-null         yes
 ipv4 192.168.11.0/24      0.0.0.0         imp-null    -                 no
 ipv4 192.168.12.0/24      0.0.0.0         imp-null    -                 no
 ipv4 192.168.66.0/24      172.20.5.5      imp-null    imp-null          no
 ipv4 192.168.77.0/24      172.20.5.5      17          imp-null         yes
 ipv4 192.168.121.0/24     172.20.5.5      imp-null    imp-null          no
 PE1#

Similar view happens in PE2.

From P1 that is our P router. We only care about LDP and ISIS

P1# 
 P1# show mpls table 
  Inbound Label  Type  Nexthop         Outbound Label  
 
 16             LDP   192.168.66.101  implicit-null   
  17             LDP   192.168.77.101  implicit-null   
 P1# show mpls ldp neighbor 
 AF   ID              State       Remote Address    Uptime
 ipv4 172.20.5.1      OPERATIONAL 172.20.5.1      02:23:55
 ipv4 172.20.5.2      OPERATIONAL 172.20.5.2      02:21:01
 P1# 
 P1# show isis neighbor 
 Area ISIS:
   System Id           Interface   L  State        Holdtime SNPA
   PE1                 ens6        2  Up            28       2020.2020.2020
   PE2                 ens7        2  Up            29       2020.2020.2020
 P1# 
 P1# show ip route
 Codes: K - kernel route, C - connected, S - static, R - RIP,
        O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
        T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
        F - PBR, f - OpenFabric,
        > - selected route, * - FIB route, q - queued, r - rejected, b - backup
 K>* 0.0.0.0/0 [0/1024] via 192.168.121.1, ens5, src 192.168.121.253, 02:24:45
 I>* 172.20.5.1/32 [115/20] via 192.168.66.101, ens6, label implicit-null, weight 1, 02:24:04
 I>* 172.20.5.2/32 [115/20] via 192.168.77.101, ens7, label implicit-null, weight 1, 02:21:39
 C>* 172.20.5.5/32 is directly connected, lo, 02:24:45
 I   192.168.66.0/24 [115/20] via 192.168.66.101, ens6 inactive, weight 1, 02:24:04
 C>* 192.168.66.0/24 is directly connected, ens6, 02:24:45
 I   192.168.77.0/24 [115/20] via 192.168.77.101, ens7 inactive, weight 1, 02:21:39
 C>* 192.168.77.0/24 is directly connected, ens7, 02:24:45
 C>* 192.168.121.0/24 is directly connected, ens5, 02:24:45
 K>* 192.168.121.1/32 [0/1024] is directly connected, ens5, 02:24:45
 P1#

So as usual, let’s try to test connectivity. Will ping from CE1 (connected to PE1) to CE3 (connected to PE2) that belong to the same VRF vrf_cust1.

First of all, I had to modify iptables in my host to avoid unnecessary NAT (iptables masquerade) between CE1 and CE3.

# iptables -t nat -vnL LIBVIRT_PRT --line-numbers
 Chain LIBVIRT_PRT (1 references)
 num   pkts bytes target     prot opt in     out     source               destination         
 1       15  1451 RETURN     all  --  *      *       192.168.77.0/24      224.0.0.0/24        
 2        0     0 RETURN     all  --  *      *       192.168.77.0/24      255.255.255.255     
 3        0     0 MASQUERADE  tcp  --  *      *       192.168.77.0/24     !192.168.77.0/24      masq ports: 1024-65535
 4       18  3476 MASQUERADE  udp  --  *      *       192.168.77.0/24     !192.168.77.0/24      masq ports: 1024-65535
 5        0     0 MASQUERADE  all  --  *      *       192.168.77.0/24     !192.168.77.0/24     
 6       13  1754 RETURN     all  --  *      *       192.168.122.0/24     224.0.0.0/24        
 7        0     0 RETURN     all  --  *      *       192.168.122.0/24     255.255.255.255     
 8        0     0 MASQUERADE  tcp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
 9        0     0 MASQUERADE  udp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
 10       0     0 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24    
 11      24  2301 RETURN     all  --  *      *       192.168.11.0/24      224.0.0.0/24        
 12       0     0 RETURN     all  --  *      *       192.168.11.0/24      255.255.255.255     
 13       0     0 MASQUERADE  tcp  --  *      *       192.168.11.0/24     !192.168.11.0/24      masq ports: 1024-65535
 14      23  4476 MASQUERADE  udp  --  *      *       192.168.11.0/24     !192.168.11.0/24      masq ports: 1024-65535
 15       1    84 MASQUERADE  all  --  *      *       192.168.11.0/24     !192.168.11.0/24     
 16      29  2541 RETURN     all  --  *      *       192.168.121.0/24     224.0.0.0/24        
 17       0     0 RETURN     all  --  *      *       192.168.121.0/24     255.255.255.255     
 18      36  2160 MASQUERADE  tcp  --  *      *       192.168.121.0/24    !192.168.121.0/24     masq ports: 1024-65535
 19      65  7792 MASQUERADE  udp  --  *      *       192.168.121.0/24    !192.168.121.0/24     masq ports: 1024-65535
 20       0     0 MASQUERADE  all  --  *      *       192.168.121.0/24    !192.168.121.0/24    
 21      20  2119 RETURN     all  --  *      *       192.168.24.0/24      224.0.0.0/24        
 22       0     0 RETURN     all  --  *      *       192.168.24.0/24      255.255.255.255     
 23       0     0 MASQUERADE  tcp  --  *      *       192.168.24.0/24     !192.168.24.0/24      masq ports: 1024-65535
 24      21  4076 MASQUERADE  udp  --  *      *       192.168.24.0/24     !192.168.24.0/24      masq ports: 1024-65535
 25       0     0 MASQUERADE  all  --  *      *       192.168.24.0/24     !192.168.24.0/24     
 26      20  2119 RETURN     all  --  *      *       192.168.23.0/24      224.0.0.0/24        
 27       0     0 RETURN     all  --  *      *       192.168.23.0/24      255.255.255.255     
 28       1    60 MASQUERADE  tcp  --  *      *       192.168.23.0/24     !192.168.23.0/24      masq ports: 1024-65535
 29      20  3876 MASQUERADE  udp  --  *      *       192.168.23.0/24     !192.168.23.0/24      masq ports: 1024-65535
 30       1    84 MASQUERADE  all  --  *      *       192.168.23.0/24     !192.168.23.0/24     
 31      25  2389 RETURN     all  --  *      *       192.168.66.0/24      224.0.0.0/24        
 32       0     0 RETURN     all  --  *      *       192.168.66.0/24      255.255.255.255     
 33       0     0 MASQUERADE  tcp  --  *      *       192.168.66.0/24     !192.168.66.0/24      masq ports: 1024-65535
 34      23  4476 MASQUERADE  udp  --  *      *       192.168.66.0/24     !192.168.66.0/24      masq ports: 1024-65535
 35       0     0 MASQUERADE  all  --  *      *       192.168.66.0/24     !192.168.66.0/24     
 36      24  2298 RETURN     all  --  *      *       192.168.12.0/24      224.0.0.0/24        
 37       0     0 RETURN     all  --  *      *       192.168.12.0/24      255.255.255.255     
 38       0     0 MASQUERADE  tcp  --  *      *       192.168.12.0/24     !192.168.12.0/24      masq ports: 1024-65535
 39      23  4476 MASQUERADE  udp  --  *      *       192.168.12.0/24     !192.168.12.0/24      masq ports: 1024-65535
 40       0     0 MASQUERADE  all  --  *      *       192.168.12.0/24     !192.168.12.0/24     
#


# iptables -t nat -I LIBVIRT_PRT 13 -s 192.168.11.0/24 -d 192.168.23.0/24 -j RETURN
# iptables -t nat -I LIBVIRT_PRT 29 -s 192.168.23.0/24 -d 192.168.11.0/24 -j RETURN

Ok, staring pinging from CE1 to CE3:

vagrant@CE1:~$ ping 192.168.23.102
 PING 192.168.23.102 (192.168.23.102) 56(84) bytes of data.

No good. Let’s check what the next hop, PE1, is doing. It seem it is sending the traffic double encapsulated to P1 as expected

root@PE1:/home/vagrant# tcpdump -i ens8
...
20:29:16.648325 MPLS (label 17, exp 0, ttl 63) (label 80, exp 0, [S], ttl 63) IP 192.168.11.102 > 192.168.23.102: ICMP echo request, id 2298, seq 2627, length 64
20:29:17.672287 MPLS (label 17, exp 0, ttl 63) (label 80, exp 0, [S], ttl 63) IP 192.168.11.102 > 192.168.23.102: ICMP echo request, id 2298, seq 2628, length 64
...

Let’s check next hop, P1. I can see it is sending the traffic to PE2 doing PHP, so removing the top label (LDP) and only leaving the BGP label:

root@PE2:/home/vagrant# tcpdump -i ens8
...
20:29:16.648176 MPLS (label 80, exp 0, [S], ttl 63) IP 192.168.11.102 > 192.168.23.102: ICMP echo request, id 2298, seq 2627, length 64
20:29:17.671968 MPLS (label 80, exp 0, [S], ttl 63) IP 192.168.11.102 > 192.168.23.102: ICMP echo request, id 2298, seq 2628, length 64
...

But then PE2 is not sending anything to CE3. I can’t see anything in the links:

root@CE3:/home/vagrant# tcpdump -i ens6
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on ens6, link-type EN10MB (Ethernet), capture size 262144 bytes
 20:32:03.174796 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e2:cb:54.8001, length 35
 20:32:05.158761 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e2:cb:54.8001, length 35
 20:32:07.174742 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:e2:cb:54.8001, length 35

I have double-checked the configs. All routing and config looks sane in PE2:

vagrant@PE2:~$ ip route
 default via 192.168.121.1 dev ens5 proto dhcp src 192.168.121.31 metric 1024 
 172.20.5.1  encap mpls  16 via 192.168.77.102 dev ens8 proto isis metric 20 
 172.20.5.5 via 192.168.77.102 dev ens8 proto isis metric 20 
 192.168.66.0/24 via 192.168.77.102 dev ens8 proto isis metric 20 
 192.168.77.0/24 dev ens8 proto kernel scope link src 192.168.77.101 
 192.168.121.0/24 dev ens5 proto kernel scope link src 192.168.121.31 
 192.168.121.1 dev ens5 proto dhcp scope link src 192.168.121.31 metric 1024 
 vagrant@PE2:~$ 
 vagrant@PE2:~$ ip -4 a
 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet 172.20.5.2/32 scope global lo
        valid_lft forever preferred_lft forever
 2: ens5:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.121.31/24 brd 192.168.121.255 scope global dynamic ens5
        valid_lft 2524sec preferred_lft 2524sec
 3: ens6:  mtu 1500 qdisc pfifo_fast master vrf_cust1 state UP group default qlen 1000
     inet 192.168.23.101/24 brd 192.168.23.255 scope global ens6
        valid_lft forever preferred_lft forever
 4: ens7:  mtu 1500 qdisc pfifo_fast master vrf_cust2 state UP group default qlen 1000
     inet 192.168.24.101/24 brd 192.168.24.255 scope global ens7
        valid_lft forever preferred_lft forever
 5: ens8:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     inet 192.168.77.101/24 brd 192.168.77.255 scope global ens8
        valid_lft forever preferred_lft forever
 vagrant@PE2:~$ 
 vagrant@PE2:~$ 
 vagrant@PE2:~$ 
 vagrant@PE2:~$ 
 vagrant@PE2:~$ ip -M route
 16 as to 16 via inet 192.168.77.102 dev ens8 proto ldp 
 17 via inet 192.168.77.102 dev ens8 proto ldp 
 18 via inet 192.168.77.102 dev ens8 proto ldp 
 vagrant@PE2:~$ 
 vagrant@PE2:~$ ip route show table 10
 blackhole default 
 192.168.11.0/24  encap mpls  16/80 via 192.168.77.102 dev ens8 proto bgp metric 20 
 broadcast 192.168.23.0 dev ens6 proto kernel scope link src 192.168.23.101 
 192.168.23.0/24 dev ens6 proto kernel scope link src 192.168.23.101 
 local 192.168.23.101 dev ens6 proto kernel scope host src 192.168.23.101 
 broadcast 192.168.23.255 dev ens6 proto kernel scope link src 192.168.23.101 
 vagrant@PE2:~$ 
 vagrant@PE2:~$                       
 vagrant@PE2:~$ ip vrf      
 Name              Table
 vrf_cust1           10
 vrf_cust2           20
 vagrant@PE2:~$ 

root@PE2:/home/vagrant# sysctl -a | grep mpls
 net.mpls.conf.ens5.input = 0
 net.mpls.conf.ens6.input = 0
 net.mpls.conf.ens7.input = 0
 net.mpls.conf.ens8.input = 1
 net.mpls.conf.lo.input = 0
 net.mpls.conf.vrf_cust1.input = 0
 net.mpls.conf.vrf_cust2.input = 0
 net.mpls.default_ttl = 255
 net.mpls.ip_ttl_propagate = 1
 net.mpls.platform_labels = 100000
root@PE2:/home/vagrant# 
root@PE2:/home/vagrant# lsmod | grep mpls
 mpls_iptunnel          16384  3
 mpls_router            36864  1 mpls_iptunnel
 ip_tunnel              24576  1 mpls_router
root@PE2:/home/vagrant#

So I am a bit puzzled the last couple of weeks about this issue. I was thinking that iptables was fooling me again and was dropping the traffic somehow but as far as I can see. PE2 is not sending anything and I dont really know how to troubleshoot FRR in this case. I have asked for help in the FRR list. Let’s see how it goes. I think I am doing something wrong because I am not doing anything new.

La Viña

I was getting many video recommendations about this type of cheesecake. So I gave it a go:

Ingredients:

4 big eggs
500g cream cheese
250g double cream
200g sugar (I used 150g)
1 tsp plain flour

Process:

Pre-heat oven at 200C
Mix cream cheese and sugar.
Add eggs and mix
Add double cream and mix
Add flour and mix
Use two wet pieces of baking paper into a round mold.
Pour the mix into the mold
Bake for 40 minutes or until quite brown on top.
Let it cool down for a bit and then put into the fridge for a couple of hours.

The recipe is quite easy and quick. To be honest, it tastes good, even better the next day. Best cheesecake ever? Don’t care, but I need to compare with the cheesecake I used to bake.

Turkish Veggie Pizza

To be honest, I dont know how to translate it, but I found the video randomly and gave it a go. Quite happy with the result.

Ingredients:

4 potatoes
1 leek
2 carrots
100 grams of spinach
1 red pepper
4 medium eggs
4 tablespoons of olive oil,
4 tablespoons of milk
4 tablespoons of flour
1/2 pack of baking powder
2 teaspoons salt, peppe
50 grams of grated cheese
Sesame seeds

Process:

1) Slice the leeks, red pepper and spinach. The add the potatoes and carrots grated. Mix all together.
2) In a bowl, whisk the eggs with the oil. Add the flour and whisk. Add the milk and whisk. Add the baking powder and whisk. Add the salt and pepper and whisk.
3) Pour the butter into the veggie try. Mix everything. Try to flat out the mix as much as you can.
4) Put the try in a pre-heat oven at 180C for 25 minutes.
5) Remove from the oven and add the cheese and sesame seeds. Put back in the oven until cheese forms a crust.
6) Remove from oven and let is cool down for a bit. Then cut in squares.

Mine was a bit thick, but nonetheless, very good!

Ark

I fancied something light to read. Oh, how enjoyed those memories when I wished to be an archaeologist like Indy. It was like watching the movie again. I didnt care I knew every twist. It brought me some smiles. Pity I devoured it so fast. Will get to the last one at some point.

Flow

I had this book in the pipeline after reading “mindset”. I dont know but some part of me always think that I am going to find my universal solution in a book. The good thing, the other part knows that is not possible. We are happy at the end. My goal is to reach a stable state of “contentment” and resilience enough to weather anything found throw life. Contentment is enjoying (different from pleasure) what I do: working, cooking, reading, sport, etc. In balance. The books explain the search for happiness in our world. How the materialistic approach doesnt work and why and how some people reach it. You need a challenge, effort, instructions, goals, feedback, etc. One quite important thing is the challenge has to be possible with our skills. So we can improve our skills and grow. If it is too much, you may not even try and if you try, you will fell worse. This is very important in the work environment where all of us spent most of our time. How would be your life if you enjoy your work? I have reached a point, that enjoying is the most important point. And yes, money is important, but is not all. It is a balance. But not all is work, so finding meaning outside work is important too. So as the author says, the goal is to have a “flow” life. There is no work-life, family-life. There is just one life. And we need to find the way to enjoy our work (life)

I was quite surprised with the section about “The Waste of Free Time”, just two pages, but hit me hard. How eager we are for having free time but then we dont use it properly. It is mainly for the entertainment industry benefit.

“The future will belong not only to the educated, but to who is educated to use her/his leisure wisely”.

Another section very close to me is “Solitude”. It remind me to a Rafael Santandreu book. In a society/world where everything has to be connected. To be alone, looks like a recipe for disaster. But it doesnt have to be that way. I am in that path. For that I think it is very important to put order in your mind, and avoid “chaos”. Again, it is putting your goals, getting feedback, instructions, etc. It is your meaning.

Gnocchi

I wanted to make a gnocchi dish for some time. I had a recipe at hand so I went for it. I used this video as a based for the gnocchi

Gnocchi in mushroom and asparagus sauce.

Gnocchi:

1kg of boiled potatoes
2 medium eggs
salt/pepper
20g Parmesan cheese
150g plain flour plus dusting

Mushroom and asparagus sauce:

a couple of garlic cloves
200g mushrooms
150g asparagus
1/2 cup of white wine
1 cup of chicken stock
Parmesan cheese
optional: lemon zest

Gnocchi Process:

1) You can boil or bake the potatoes. Let it cool down a bit until you can manage with your hands. Keep the water for next steps.
2) Put in a bowl, use a potato smasher to break them a bit.
3) Add the eggs, salt, pepper and cheese. Mix a bit .and then
4) Add the flour. You should get a ball. Be sure everything is properly combined.
5) Get a saucepan with boiling water and salt from the first step.
6) In your work surface, dust it with flour, device your dough in several parts. Roll each one until you have a flaute form. Add flour if it sticks too much. Cut them in small pieces
7) Boil the gnocchi until the float. Remove from water and let them cool down in a try (put some olive oil so they dont stick to it)

Sauce Process:

1) Fry the garlic in a sauce pan
2) Add chopped mushrooms and asparagus. Salt and pepper. And other herbs you fancy
3) Once the veggies are fried (medium-high heat), add the wine. It should evaporate quickly and form some brown bits in the pan.
4) reduce heat, add the chicken stock. Stir from time to time until thickens a bit.
5) Add the gnocchi to the sauce, with some pasta water. Be sure they are fully coated.
6) You can fry a bit the gnocchi, be sure there is sauce left!
7) Optional, add a bit of lemon zest.
7) Remove pan from the heat, and let cool down for a bit. Ready to eat!

I was surprised by the gnocchi, they were much better than the ones you buy. It was like small dumplings balls! Very tasty.

Lamingtons

It seems this is a typical Australian desert.

Sponge:

250g golden caster sugar
250g self raising flour
1/2 tsp salt
125g butter softened
3 medium eggs
3 tsb milk

Filling:

2 tsp icing sugar
250g double cream
200g jam (any red berries are nice)

Coating:

50g cocoa powder
200g icing sugar
80g butter melted
200g milk
200g desiccated coconut

Process:

1) Pre-heat oven at 200C. Grease a baking try
2) In a bowl, cream the butter and sugar.
3) Add the eggs, one at each time. Only add the next one when is everything is properly combined
4) Add the flour, milk and salt until fully combined. Pour the mix into the baking try.
5) Bake for 20 minutes or until golden. Use a knife and be sure it comes up clean. Set aside to cool down
6) Filling: In a bowl, sift the icing sugar and add the double cream. Mix with a hand blender until thick.
7) Once the sponge is cold, slice it horizontally. In one side, pour the filling (use a spoon through boiling water to help you to spread it easier).
8) On top of the filling, pour the jam, use again the spoon through boiling water.
9) Sandwich the sponge back together. Let is chill in the fridge while preparing the coating.
10) Whisk together the melted butter and milk in a bowl. In other bowl, sieve the cocoa powder and icing sugar. Bit by bit, add the cocoa/sugar mix to the butter mix. Whisk until no lumps.
11) Cut the sponge in squares, dip each square in the cocoa mix until properly coated. Then roll in the desiccated coconut. Put aside.
12) Once all squares are done, sprinkle some more coconut.
13) Let the cocoa coating to settle for 1h. But next day are even better!

Cornbread

This is something I wanted to try from sometime. It is not my typical bread. I think I tried first time in a BBQ restaurant. I followed this recipe.

Ingredients

120g butter
1/2 bunch fresh sage + 1/2 bunch fresh thyme ( I used dried herbs I didnt have fresh)
180g plain flour
50g sugar
3 heap tablespoons light brown sugar
6g fine sea salt
1 tablespoon baking powder
180g yellow fine corn meal
2 eggs
350ml buttermilk ( I used milk + 2 tsp of white wine vinegar as I didnt have this)

Process

1- Brown the butter in a hot pan. Retire add the herbs.

Preheat the oven at 200C

2- Mix all dry ingredients.

3- Whisk the eggs, add the buttermilk, then add the butter.

4- Pour the wet mixture into the dry mixture. Combine until there are no lumps.

5- Pour the mix in a dish oven (add a bit of butter to the surface so it doesnt stick)

6- Put in the oven for 25 minutes, top a bit brown. Be sure if you use a knife, it comes out clean. Then it is read.

7- Let it cool down and ready to eat!

Linux+MPLS-Part3

Continuation of the second part, this time we want to test VRF-lite.

Again, I am following the author post but adapting it to my environment using libvirt instead of VirtualBox and Debian10 as VM. All my data is here.

This is the diagram adapted to my lab:

After updating Vagrantfile and provisioning script, I “vagrant up”. The 6 VMs dont take long to boot up so it is a good thing.

The provisioning script is mainly for configuration of PE1 and PE2 . This is a bit more detail:

    # enabling ipv4 forwarding (routing)
    sudo sysctl net.ipv4.ip_forward=1

    # add loopback (not used in lab3)
    sudo ip addr add 172.20.5.$self/32 dev lo

    # removing ip in link between pe1-pe2 as we will setup a trunk with two vlans.
    sudo ip addr del 192.168.66.10$self/24 dev ens8

    # creating two vlans 10 (ce1,ce3) and 20 (ce2, ce4)
    sudo ip link add link ens8 name vlan10 type vlan id 10
    sudo ip link add link ens8 name vlan20 type vlan id 20

    # assign IP to each vlan
    sudo ip addr add 172.30.10.10$self/24 dev vlan10
    sudo ip addr add 172.30.20.10$self/24 dev vlan20

    # turn up each vlan as by default are down
    sudo ip link set vlan10 up
    sudo ip link set vlan20 up

    # create two routing tables with a null route
    sudo ip route add blackhole 0.0.0.0/0 table 10
    sudo ip route add blackhole 0.0.0.0/0 table 20

    # create two VRFs and assign one table (created above) to each one
    sudo ip link add name vrf_cust1 type vrf table 10
    sudo ip link add name vrf_cust2 type vrf table 20

    # assign interfaces to the VRFs            // ie. PE1:
    sudo ip link set ens6 master vrf_cust1     // interface to CE1
    sudo ip link set vlan10 master vrf_cust1   // interface to PE2-vlan10

    sudo ip link set ens7 master vrf_cust2     // interface to CE2
    sudo ip link set vlan20 master vrf_cust2   // interface to PE2-vlan20

    # turn up VRFs
    sudo ip link set vrf_cust1 up
    sudo ip link set vrf_cust2 up

    # add static route in each VRF routing table to reach the opposite CE
    sudo ip route add 192.168.$route1.0/24 via 172.30.10.10$neighbor table 10
    sudo ip route add 192.168.$route2.0/24 via 172.30.20.10$neighbor table 20

Check the status of the VRFs in PE1:

vagrant@PE1:/vagrant$ ip link show type vrf
 8: vrf_cust1:  mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
     link/ether c6:b8:f2:3b:53:ed brd ff:ff:ff:ff:ff:ff
 9: vrf_cust2:  mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 62:1c:1d:0a:68:3d brd ff:ff:ff:ff:ff:ff
 vagrant@PE1:/vagrant$ 
 vagrant@PE1:/vagrant$ ip link show vrf vrf_cust1
 3: ens6:  mtu 1500 qdisc pfifo_fast master vrf_cust1 state UP mode DEFAULT group default qlen 1000
     link/ether 52:54:00:6f:16:1e brd ff:ff:ff:ff:ff:ff
 6: vlan10@ens8:  mtu 1500 qdisc noqueue master vrf_cust1 state UP mode DEFAULT group default qlen 1000
     link/ether 52:54:00:33:ab:0b brd ff:ff:ff:ff:ff:ff
 vagrant@PE1:/vagrant$

So let’s test if we can ping from CE1 to CE3:

Ok, if fails. I noticed that PE1 sees the packet from CE1… but the source IP is not the expected one (11.1 is the host/my laptop). And the packet reaches to PE2 with the same wrong source IP and then to CE3. In CE3 the ICMP reply is sent to 11.1, to it never reaches CE1.

The positive thing is that VRF lite seems to work.

I double checked all IPs, routing, etc. duplicated MAC in CE1 and my laptop maybe??? I installed “net-tools” to get “arp” command and check the arp table contents in CE1. Checking the ARP request in wireshark, all was good.

Somehow, the host was getting involved…. Keeping in mind that this is a simulated network, the host has access to all “links” in the lab. Libvirt creates a bridge (switch) for each link and it adds a vnet (port) for each VM that uses it:

# brctl show 
 bridge name    bridge id       STP enabled interfaces
 virbr10        8000.525400b747b0   yes     vnet27
                                            vnet30
 virbr11        8000.5254006e5a56   yes     vnet23
                                            vnet31
 virbr12        8000.525400dd521a   yes     vnet19
                                            vnet21
 virbr3        8000.525400a38db1   yes     vnet16
                             vnet18
                             vnet20
                             vnet24
                             vnet26
                             vnet28
 virbr8        8000.525400de61f2   yes     vnet17
                                           vnet22
 virbr9        8000.525400e2cb54   yes     vnet25
                                           vnet29

“.1” is always the host but It was clear my routing was correct in all devices. I remembered that I had some issues during the summer when I was playing with containers/docker and doing some routing…. so I checked iptables….

I didnt have iptables in the VMs… but as stated earlier, the host is connected to all “links” used between the VMs. There is no real point-to-point link.

# iptables -t nat -vnL --line-numbers
...
Chain LIBVIRT_PRT (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1       11   580 RETURN     all  --  *      *       192.168.11.0/24      224.0.0.0/24        
2        0     0 RETURN     all  --  *      *       192.168.11.0/24      255.255.255.255     
3        0     0 MASQUERADE  tcp  --  *      *       192.168.11.0/24     !192.168.11.0/24      masq ports: 1024-65535
4       40  7876 MASQUERADE  udp  --  *      *       192.168.11.0/24     !192.168.11.0/24      masq ports: 1024-65535
5       16  1344 MASQUERADE  all  --  *      *       192.168.11.0/24     !192.168.11.0/24     
6       15   796 RETURN     all  --  *      *       192.168.24.0/24      224.0.0.0/24        
7        0     0 RETURN     all  --  *      *       192.168.24.0/24      255.255.255.255     
8        0     0 MASQUERADE  tcp  --  *      *       192.168.24.0/24     !192.168.24.0/24      masq ports: 1024-65535
9       49  9552 MASQUERADE  udp  --  *      *       192.168.24.0/24     !192.168.24.0/24      masq ports: 1024-65535
10       0     0 MASQUERADE  all  --  *      *       192.168.24.0/24     !192.168.24.0/24     



# iptables-save -t nat
# Generated by iptables-save v1.8.7 on Sun Feb  7 12:06:09 2021
*nat
:PREROUTING ACCEPT [365:28580]
:INPUT ACCEPT [143:14556]
:OUTPUT ACCEPT [1617:160046]
:POSTROUTING ACCEPT [1390:101803]
:DOCKER - [0:0]
:LIBVIRT_PRT - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-4bd17cfa19a8 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-43481af25965 -j MASQUERADE
-A POSTROUTING -j LIBVIRT_PRT
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-4bd17cfa19a8 -j RETURN
-A DOCKER -i br-43481af25965 -j RETURN
-A LIBVIRT_PRT -s 192.168.11.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.11.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.11.0/24 ! -d 192.168.11.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.11.0/24 ! -d 192.168.11.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.11.0/24 ! -d 192.168.11.0/24 -j MASQUERADE
-A LIBVIRT_PRT -s 192.168.24.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.24.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.24.0/24 ! -d 192.168.24.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.24.0/24 ! -d 192.168.24.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.24.0/24 ! -d 192.168.24.0/24 -j MASQUERADE

Ok, it seems the traffic form 192.168.11.0 to 192.168.23.0 is NAT-ed (masquerade in iptables). So makes sense that I see the traffic as 11.1 in PE1. Let’s remove that:

# iptables -t nat -D LIBVIRT_PRT -s 192.168.11.0/24 ! -d 192.168.11.0/24 -j MASQUERADE

Test again pinging from CE1 to CE3:

So it works properly, we can see the the correct IPs in every hop: PE1, PE2 and CE3.

So it seems this is a built-in behaviour in libvirt. I need to find out how to “fix” this behaviour whenever I do “vagrant up”.

Canelones de espinacas

Pasta bolognesa was my favourite dish as a kid. We had it I think once every week. Canelones was another big one but my mum didnt cook it that often as it was more time consuming assembling the canelones and preparing the white sauce.

Anyway, I wanted to try the spinach version so I found this recipe that looked nice. I used lasagna sheets cut in two for the canelones and worked nicely.

Ingredients for filling

12 lasagna sheets
250g frozen spinach (put in boiling water to unfreeze)
1 chopped onion
1 piece of garlic
1 boiled egg
20g of pine nuts
a bit of grated parmesan cheese (or similar hard cheese)

Ingredients for bechamel sauce:

2-3 big tsp of plain flour
25g of butter
nutmeg
salt and pepper
200ml milk
grated parmesan cheese

Process

Boil 1l water in a pan, with salt and a bit of olive oil. Once boiling, put lasagna sheet in 2-3 at each time. They should be ready in a couple of minutes (they should float). Remove and leave it rest.
In a sauce pan, heat some oil, add the onion and garlic. Until soft
Add the unfrozen spinach, keep stirring.
Add the pine nuts and the boiled egg (cut in small pieces). Salt and pepper.
Add a bit of cheese.
Retire from the heat.
In a saucepan, melt the butter, add the flour little by little until forming a paste.
Start adding the milk, again little by little stirring without stopping. This is at low-medium heat. Add nutmet, salt and pepper. Taste from time to time. Add the cheese at the end.
Once you have a sauce that is not very liquid neither thick, we are ready to assemble everything.
Pre-heat oven at 200C
Put some of the bechamel in the spinach mix and stir.
Prepare an oven dish. Put some butter or bechamel sauce in the dish to avoid stickiness.
Take the cold lasagne sheets, cut in two squares. In each square put a spoon of the spinach mix and roll like a canelone. Put in the oven dish in lines.
Once you the oven dish covered with canelones, pour the bechamel soauce on top of them. Add some grated cheese. I had some tomato sauce left over and I add it too.
Put the oven dish in the oven. 30m or until the top is grilled.
Let it cool down a bit and ready to eat!