Meta Chips – Colvore water-cooling – Google AI TPv4 – NCCL – PINS P4 – Slingshot – KUtrace

Read 1. Meta to build its own AI chips. Currently using 16k A100 GPU (Google using 26k H100 GPU). And it seems Graphcore had some issues in 2020.

Read 2. Didnt know Colovore, interesting to see how critical is actually power/cooling with all the hype in AI and power constrains in key regions (Ashburn VA…) And with proper water cooling you can have a 200kw rack! And seems they have the same power as a 6x bigger facility. Cost of cooling via water is cheaper than air-cooled.

Read 3. Google one of biggest NVIDIA GPU customer although they built TPUv4. MS uses 10k A100 GPU for training GPT4. 25k for GPT5 (mix of A100 and H100?) For customer, MS offers AI supercomputer based on H100, 400G infiniband quantum2 switches and ConnectX-7 NICs: 4k GPU. Google has A3 GPU instanced treated like supercomputers and uses “Apollo” optical circuit switching (OCS). “The OCS layer replaces the spine layer in a leaf/spine Clos topology” -> interesting to see what that means and looks like. As well, it uses NVSwitch for interconnect the GPUs memories to act like one. As well, they have their own (smart) NICS (DPU data processing units or infrastructure processing units IPU?) using P4. Google has its own “inter-server GPU communication stack” as well as NCCL optimizations (2016! post).

Read4: Via the P4 newletter. Since Intel bought Barefoot, I kind of assumed the product was nearly dead but visiting the page and checking this slides, it seems “alive”. Sonic+P4 are main players in Google SDN.

 “Google has pioneered Software-Defined Networking (SDN) in data centers for over a decade. With the open sourcing of PINS (P4 Integrated Network Stack) two years ago, Google has ushered in a new model to remotely configure network switches. PINS brings in a P4Runtime application container to the SONiC architecture and supports extensions that make it easier for operators to realize the benefits of SDN. We look forward to enhancing the PINS capabilities and continue to support the P4 community in the future”

Read5: Slingshot is another switching technology coming from Cray supercomputers and trying to compete with Infiniband. A 2019 link that looks interesting too. Paper that I dont thik I will be able to read neither understand.

Read6: ISC High Performance 2023. I need to try to attend one of these events in the future. There are two interesting talks although I doubt they will provide any online video or slides.

– Talk1: Intro to Networking Technologies for HPC: “InfiniBand (IB), High-speed Ethernet (HSE), RoCE, Omni-Path, EFA, Tofu, and Slingshot technologies are generating a lot of excitement towards building next generation High-End Computing (HEC) systems including clusters, datacenters, file systems, storage, cloud computing and Big Data (Hadoop, Spark, HBase and Memcached) environments. This tutorial will provide an overview of these emerging technologies, their offered architectural features, their current market standing, and their suitability for designing HEC systems. It will start with a brief overview of IB, HSE, RoCE, Omni-Path, EFA, Tofu, and Slingshot. In-depth overview of the architectural features of IB, HSE (including iWARP and RoCE), and Omni-Path, their similarities and differences, and the associated protocols will be presented. An overview of the emerging NVLink, NVLink2, NVSwitch, Slingshot, Tofu architectures will also be given. Next, an overview of the OpenFabrics stack which encapsulates IB, HSE, and RoCE (v1/v2) in a unified manner will be presented. An overview of libfabrics stack will also be provided. Hardware/software solutions and the market trends behind these networking technologies will be highlighted. Sample performance numbers of these technologies and protocols for different environments will be presented. Finally, hands-on exercises will be carried out for the attendees to gain first-hand experience of running experiments with high-performance networks”

– Talk2: State-of-the-Art High Performance MPI Libraries and Slingshot Networking: “Many top supercomputers utilize InfiniBand networking across nodes to scale out performance. Underlying interconnect technology is a critical component in achieving high performance, low latency and high throughput, at scale on next-generation exascale systems. The deployment of Slingshot networking for new exascale systems such as Frontier at OLCF and the upcoming El-Capitan at LLNL pose several challenges. State-of-the-art MPI libraries for GPU-aware and CPU-based communication should adapt to be optimized for Slingshot networking, particularly with support for the underlying HPE Cray fabric and adapter to have functionality over the Slingshot-11 interconnect. This poses a need for a thorough evaluation and understanding of slingshot networking with regards to MPI-level performance in order to provide efficient performance and scalability on exascale systems. In this work, we delve into a comprehensive evaluation on Slingshot-10 and Slingshot-11 networking with state-of-the-art MPI libraries and delve into the challenges this newer ecosystem poses.”

Read7: Slides and Video. I was aware of Dtrace (although never used it) so not sure how to compare with KUtrace. I guess I will ask Chat-GPT 🙂

Read8: Python as programming of choice for AI, ML, etc.

Read9: M$ “buying” energy from fusion reactors.

Mobile Phone + SSH server

I have tried many times to connect my mobile phones to my laptop. It always looks easy if you use M$ but with Linux I always fail, I can’t get to work MTP. So now I really want to take mainly all my pictures from a phone and be able to back them up and transfer to a new one. I dont want to use cloud services or tools from the manufacturers. I want to use old school methods. So after struggling for some time, I somehow decided to use something as old school as SSH/SCP. Android is based in linux, isn’t it? So I searched for a free SSH server app, found this one. And it worked! I managed to understand it, created my user, my mounting points, enable it… and was able to SCP all my photos from my mobile to my laptop. It worked with Samsung and Huawei.

I am pretty sure that people know have better ways to do this… but that’s me.

Chocolate Fondant

Based on this video:

Ingredients:

  • 130g dark chocolate (+70%)
  • 130g butter + extra for greasing
  • 3 eggs
  • 1 egg yolk
  • 100g caster sugar
  • 70g plain flour
  • 15g cocoa powder + extra for dusting
  • 2.5g baking powder
  • 4 aluminium moulds

Process:

  • Pre-heat oven at 180C
  • Melt chocolate and butter at “baine marie”. Let it cool down a bit to use later.
  • Grease with butter the moulds and dust the sides with cocoa powder
  • Make the “raw sabayon”: whisk the eggs and sugar until pale in color.
  • Fold the chocolate (be sure it is not too hot) into the sabayon. Be sure the mix is uniform and there are no lumps.
  • Sieve flour, cocoa and baking powder. Then add to the mix bit by bit, folding with a spatula and checking there are no lumps.
  • Fill the moulds at 90% aprox. They will raise in the oven
  • Bake the moulds at 180C for 9 minutes aprox.
  • Use a tooth stick to be sure they are still creamy inside. The idea is the chocolate should come out once opened.
  • Unmould and present on a plate with a bit of fresh mint, strawberries. Dust with a bit of icing sugar.
  • Be sure you serve it hot! (optionally you can add a ball of vanilla ice cream).

No free lunch

Just a reminder

“Every single pursuit – no matter how wonderful and exciting and glamorous it may initially seem – comes with its own brand of shit sandwich, its own lousy side effects. Everything sucks some of the time. You just have to decide what sort of suckage you’re willing to deal with.
​
So the question is not so much ‘What are you passionate about?’
​
The question is ‘What are you passionate enough about that you can endure the most disagreeable aspects of the work?’
​
Because if you love something and want something enough – whatever it is – then you don’t really mind eating the shit sandwich that comes with it.”
​
- Mark Manson
In a world of comfort one question the most important question to ask yourself is what are you willing to struggle for? You can have the pain of being lazy or the pain of putting an effort into your health and body? Which one will it be?

Sable a la poche

Or “sand in the pocket” biscuits or just super tasty biscuits with a glass of cold milk.

Wet Ingredients

  • 340gr butter at room temperature
  • 128gr icing sugar
  • 50gr egg whites
  • 1 tsp vanilla

Dry Ingredients:

  • 380gr strong white flour (bread flour)
  • 1 tsp baking powder
  • 1 tsp salt
  • Glace cherries (for decoration)

Process

  • Pre-heat oven at 180C
  • Sift dry ingredients together and set aside.
  • Cream the butter until pale and creamy. Then add the icing sugar.
  • Add the egg whites and vanilla 1/3 at the time to the butter mix until fully incorporated.
  • Add the flour to the wet ingredients. Mix until fully combined, dont overmix.
  • Using a pastry bag with a large star tip, pipe into 5-6cm wide circles into your baking tray.
  • Decorate with glace cherries
  • Bake at 180C for 15 minutes or until lightly golden and well-browned in the edges.
  • Let it cool down

Once of the best biscuits I have done. Very easy to make, quick and tasty! Sold.

Taste

I decided to buy and read this book one day while checking out some stands in a library. I am not keep of biographies of “famous” people. It rang the bell that he was an actor so I was surprised to find a book about food. Checking the cover there was a sentence that bought me “he grew up in an Italian American family that spent every night around the table”. That’s what I like, build a culture around food, preparing and enjoying it.

The book is not a recipe book, it is a history around food. It has some funny moments and more important thing, some recipes to try. And will do my best to do so.

As well, there is an important reference to “Big Night” and actually watched it this weekend. Nothing really special but I got his point about the food and the enjoyment about it. And that reminded me to the movie Chef, I liked it more.

There is another important reference to “Julia Child” who was a famous TV cook/chef/presenter. Some weeks ago having lunch with friends somebody mentioned her and a movie about her life: “Julia and Julie” while talking about cooking and nice food. And I am trying to convince myself to watch it.

A good point is the critic to people on TV tasting food and always saying that is amazing when they haven’t had the time to swallow! I always thought it was a bit fake, so I am glad I am not the only one thinking that.

I am surprised by the outcome of his recovery of tongue cancer. His metabolism and allergies were “reset”. So no more lactose issues and improved digestive system.

Anyway, it was entertaining and I hope I can take some recipes for my own repository.

Tarta de Zanahoria

This is not the typical “carrot cake”. I have some memories when my mother used to make this cake. I have tried to find the recipe in her papers a couple of times but never found it (I think). So I asked a very good friend of her about this recipe and lucky enough she had it so managed to take a picture of it. And finally this weekend I decide to try.

Ingredients:

  • 250g of sponge fingers
  • 400ml milk
  • 200g sugar
  • 150gr cocunut sherred
  • 500g peeled carrots
  • 1 tin of 1L aprox

Process:

  • Boil the carrots in a sauce pan with just a 2 fingers of water. Check from time to time that there is enough water and they dont stick to the bottom. If they do, just add some water and scrap the bottom. It is the sugar from the carrots so it is good.
  • Once the carrots at soft enough so a fork can easily go through them, remove from the heat. Remove all the water with a sieve or strain.
  • Put the carrots, milk, sugar and 100g of coconut in the food processor. Mix until all combined like a cream.
  • In the tin, cover the bottom with a layer of sponge finger. Then add some carrot cream slowly so the fingers dont move. Press a bit with your hand so all cream gets into the sponges. Add another layer of sponge finger, add another layer of carrot cream. Add a third and final layer of sponge fingers and carrot cream.
  • Put the tin in the fridge for 3-4 hours.
  • You can remove the cake from the tin if you want to present to people. Then add the rest of the coconut on top. You can add some berries on top optionally.
  • Optionally, you can just spread the rest of coconut on top of the tin.

This is the resut:

It is quite tasty and really simple. The only “difficult” ingredient maybe is the coconut shreds and find the sponge fingers.

And to be honest, looks like a carrot “tiramisu” 🙂 And although it is a nice cake somehow it is not what I have in my memory. Anyway, I will keep this recipe.

VXLAN BGP EVPN Multisite

This is a video that explains high level about EVPN Multisite. There is no really config involved. The pdf for the session “BRKDCN-2913” is easy to find and download. Although this is NXOS based, Arista has similar feature called “EVPN Gateway”:  https://www.arista.com/en/support/toi/eos-4-25-0f/14591-evpn-l3-gateway (needs registration….) Just one line really to add under the EVPN address family to change the next hop to the gateway’s address. The implementation looks much more simpler than NXOS….

This is a summary of the video:


RFC9014 … DCI EVPN Overlay defines the Layer-2 extension between two domains

section 3: decoupled gw. vland handoff with a WAN edge.
section 4: integrated gw: gw talk directly L2EVPN
multi-site (BESS version) draft-sharma-bess-multi-site-evpn. support extension of l2 and l3, uc and mc, vpns. BGW talk ebgp evpn AF.
gw mode: anycast vip (ecmp: underlay) or multipath vip (ecmp: under and overlay)
type5: re-originated.
RD: separate RD for vIP and PIP
RT: same for intra/inter dc
Border GW = EVPN GW

EVPN-IPVPN interop defines the Layer-3 extension between domains, currently lacks of EVPN to EVPN interconnects

Multisite draft combines RFC9014 and EVPN-IPVPN with EVPN to EVPN connection: https://datatracker.ietf.org/doc/html/draft-sharma-bess-multi-site-evpn-02

Use cases:
1- Compartmentalization:

  • multiple fabrics, single DC
  • control at BGW: allow extension l2,l3. Reduces remote VTEP count. Expands VTEP scale.
  • BUM packet: LS replicated only in the fabric, then BGW to the BGW in the other fabric. In no multi-site, LS replicate to ALL VTEP in the fabric.

2- Scale

  • control at BGW: Reduces remote VTEP count. Expands VTEP scale.
  • scale thhrough hierarchy: multiply vtep with sites
    up to 128 sites per multi-site domain. Up to 256 VTEP per fabric -> 32768 VTEPs

3- DC interconnect (DCI)

  • IP reachability and MTU.
    integration with legacy networks:
    hybrid cloud connectivity: extends l3 with vrf awareness.

Deeper look:
HW support only important in BGW. LS is not important.

tunnels:

  • stitched at BGW (no recirculation, hw rate)
  • intra fabric tunnel goes LS to LS or LS to BFW
  • inter fabric tunnel goes BGW to BGW
  • only BGW IP must be unique.. Fabrics are “separated”.

BGW deployment considerations:

  • 1) anycast bgw
  • – up to 6 nodes. They are not interconnected, just share ASN nothing else.. In LS or SS
  • – VIP mode: vip for tunnel stitching. foucs on scale and convergence. overlay ecpm
  • – PIP mode: for 3rd party interop. Uses PIP for tunnel stitching. Uses under and overlay Ecmp.

  • 2) vpc bgw:
  • – only 2 (because vpc, peer link). Only in LS
    – legacy network integration, attachment of fw and adcs.

NOTE: anycast and vpc must have a multi-site vip and PIP. only vpc needs an extra IP for VPC IP.
PIP needed for establishing BGP and for Designated Forwarding election (only one BGW forwards per vlan.

CP and DP:

  • As eBGP uses betweem multi-sites -> ebgp changes NH => vxlan tunnel termination and re-origination + loop prevention (as-path). Full mesh ebgp evpn between sites.
  • underlay/overlay CP deployemnt: recommended IEI (recommended) within fabric: IGP as underlay, iBGP as overlay.
  • full mesh ebgp evpn between site OR deploy RS (route-server) -> RS is in a separate AS and only does CP = eBGP RR (RFC 7947): evpn routes reflection, NH unchanged, RT rewrite!

I think this is the white paper mentioned:  https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html

Another thing, I wish it wouldnt be that painful to simulate NXOS. It is so easy spin up a lab with cEOS…..in a standard laptop..

Pollo en escabeche

I like a lot “atun en escabeche” but it is really difficult to find. And to make it, quite expensive… So during a conversation, a friend of mine, mentioned “pollo en escabeche” as vey tasty dish with similar flavour to “atun en escabeche”. So I decided to give it a go and find a recipe. This one looked very good.

Ingredients:

  • 400g chicken breast
  • 1 big onion sliced
  • 2 large carrots sliced
  • 3 cloves of garlic
    • 1 glass of white wine
    • 1 bay leaf
    • 15 pepper corns
    • 1 glass of vinegar (white wine, apple, etc)
    • 1 glass of water
    • salt + pepper

Process

  • Salt and pepper the chicken breasts.
  • Fry the chicken in a hot pan until golden outside. Put apart.
  • In the same pan, add the onions, carrots and garlic. Fry until soften, aprox 4-5 minutes.
  • Add glass of wine, bay leaf and pepper corns. Keep the high heat so the alcohol evaporates.
  • Then add the glass of vinegar and water.
  • Add the chicken back in.
  • Cook at high heat for 15 minutes so the liquid evaporates a bit and the chicken absorbs some juice.
  • Remove from the heat and pout the content of the pan into a glass dish.

Ready to eat!

Very easy, quick and tasty!

Brioche

After the positive experience with Babka, I wanted to try Brioche. I did all kneading by hand. All videos I have seen use a machine. It took me 1h aprox to do all kneading and adding the butter but it was so worth it to see that I was able to achieve a dough that stopped being sticky, smooth and glossy.

Ingredients:

  • 500g strong white flour
  • 12g fine sea salt
  • 30g caster sugar
  • 8g dry yeast
  • 6 eggs (1 egg aprox 50g)
  • 250g softened unsalted butter. Chopped in small cubes.
  • 1 egg yolk for glazing

Process:

  • Put flour, salt and sugar in a bowl and mix. Then add the yeast and mix well.
  • In another bowl, break all eggs, whisk them and then add them to the flour making a well in the center.
  • Mix all ingredientes with one hand, once it comes together, tip the dough on the work surface. Dont add any flour!
  • Use one hand for kneading and the other for the scraper.
  • With the heel of your hand, push the dough into the table and stretch and tear until it stops being sticky. Use the scrapper to get the dough together and clean your dirty hand. Aprox 10 minutes.
  • Let the dough rest 5 minutes
  • Now, add slowly the butter to the dough in small batches. I did 6 batches… So stretch the dough a bit and add some cubes of butter, try to “integrate” them in the dough with your dirty hand. Always keep the clean hand with the scraper. Then start to stretch/tear the dough. It will become a bit “wet” with the butter but will come together again, and will not stick to the surface after some minutes. It is amazing to see that happening. Repeat the process until all butter is added in the dough.
  • At the end, the dough should no sticky to the table, should be glossy, smooth and elastic.
  • Put the dough back in bowl. Cover and leave at room temperature for 2h or until has doubled in size.
  • Knock back the dough, cover again and put back in the fridge until next day.
  • Next day, line a tin with baking paper or grease it. I used a big tin. You can divide the dough in two if you have smaller tins.
  • Take the dough and divide in 3 pieces. Aprox 360g each. Round each piece gently. Then roll each piece a bit longer than your tin.
  • Time for the “plait”. Join the three ends together, then plait by lifting the middle strand and putting the right strand in the middle. The “lift” strand moves to the right and release it. Now lift again the middle strand, take the left strand and put in the middle. The “light” strand moves to the left side and release. Repeat again the same process, lift middle, move right, lift middle, move left. Until you dont have more to braid. Pinch the ends.
  • Place the brioche in the tin and leave to rise for 2h aprox or reaches the top of the tin.
  • Pre-heat the oven at 180C.
  • Glaze the brioche with the egg yolk.
  • Put the brioche in the oven and lightly spritz the oven with a water spray.
  • Bake for 20 minutes. Then remove from the tin and bake another 5 minutes or until golden brown.
  • Take from the oven and let it cool for a couple of minutes (if you can)

And really happy with the result!

As mentioned earlier, I was quite surprised that I managed again to tame an enriched dough. Still I can improve it:

  • I did 4 strands but I have realized that 3 is better and easier to braid…..
  • I should have used more egg yolk for the glaze.