Day: 28 October 2023

This is a conference about networks that I was interested and I finally got some emails with the presentations. They are mainly from Meta.

Meta’s Network Journey to Enable AI: video – second part interesting.

• AI fabric (backend: gpu to gpu) hanging from DC fabric.
• SPC (Space, Power, Cooling)
• Fiber, Automation
• RDMA requires (lossless, low-latency, in-order) -> ROCEv2(Ethernet) or IB
• Servers have 8x400G to TOR. Tor 400G to Spines
• 1xAI zone per DH. 1xDC has several DHs.
• Oversubscribed between zones, eBGP, ECMP.

Scaling RoCE Networks for AI Training: video — Really really good.

• RMDA/IB used for long time in Research.
• Training: learning a new capability from existing data (focus of the video)
• Inference: Applying this capability to new data (real time)
• Distributed training for complex models. GPU to GPU sync -> High BW and low/predictable latency.
• ROCEv2 with (tuned) PFC/ECN. TE + ECMP (flow multplexing)
• Oversubscription is fine in spine (higher layer)
• Challenges: Load balancing (elefant flows), Slow receivers/back pressure, packet loss from L1 issues (those flapping links, faulty optics, cables, etc xD), debugging (find jobs failures)

Traffic Engineering for AI Training Networks : video – interesting both parts.

• Non-blocking. RTSW=TOR. CTSW=Spine. Fat-Tree Architecture. 2xServer per rack. 1xserver=8xGPU. CTSW=16 downlinks -> 16 uplinks. Up to 208 racks?
• ROCE since 2020. CTSW are high redix and deep buffer switches.
• AI Workload challenges: low entropy (flow repetitive, predictable), bursty, high intensity elephant flows.
• SW based TE: dynamic routing adapted on real-time. Adaptive job placement. Controller (stateless)
• Data plane: Overlay (features from broadcom chips) and Underly (BGP)
• Flow granularity: nic to host flow.
• Handle network failures with minimum convergence time. Backdoor channel with inhouse protocol.
• Simulation platform. NCCL benchmark.

Networking for GenAI Training and Inference Clusters: video Super Good!

• Recommendation Model: training 100GFlops/interation. inference: few GFlops/s for 100ms latency.
• LLM: training 1PetaFlops/sentence (3 orders magnitude > recommendation), inference: 10PF/s for 1sec time-to-first token. +10k GPUs for training. Distributed inferencce. Need Compute too.
• LLama2 70Billion tokens -> 1.7M hours of GPU. IB 200G per GPU, 51.2 TB/s bisection bw. 800 ZetaFlops. 2 Trillion dataset. 2k A100 GPUs. As well, used ROCEv2 (LLama2 34B).
• +30 ExaFlops (30% of H100 GPUs fp8 peak) + LLama65B training < 1day.
• Massive cluster: 32k GPUs! Model Parallelism.
• LLM inference: dual-edge problem. Prefill large messages (High BW) + Decode small messages (latency sensitive).
• Scale out (-bw, large domain. Scalable RDMA (IB or Ethernet), data parallel traffic) + Scale up (+BW, smaller domain. NVLink 400G, model parallel traffic)
• 32k GPU. TOR (252), Spine (18), AGG (18). 3 levels. Oversubscription Spine-Agg 7:1. 8 clusters. 252 racks per cluster. 16 GPUs per rack (8x252x16=32k GPUs). ROCEv2!
• Model Parallelism harder for computation. Model Parallel traffic: all-reduced/all-to-all, big messages (inside cluster = sclae-up). Data Parallel traffic: all-gather & reduce-scatter (between cluster = scale-out, NVLink)
• Challenges: Latency matters more than ranking. Reliability !!!!!
• LLM inference needs a fabric.

Scale out vs scale up: storage DB

scale up (vertical): more bw(links), more storage, etc

scale out (horizontal): distribute load into different devices

Network Observability for AI/HPC Training Workflows: video

• ROCET: Automating RDMA metric collection and analysis for GPU training. Info from hosts/nics and switches.
• Report: out-of-sequence, nic flaps, local ack timeouts.
• PARAM + pytorch. Chakra.

From ESNOG (although I am subscribed, I dont really receive notifications…) I saw this presentation, and then I found the main video reference.

BUM: 1.5Mbps -> that goes to all customer ports!! Small devices can’t cope with that.

Ratelimiting for BUM ingress/egress

Originally IXP is VPLS. Issue with long lasting flows / long lasting MACs

Solution: EVPN + ProxyARP/ND Agent.

I know EVPN is mainly for Datacenters but it is interesting the move to IXP. Although I remember reading about EVPN into Cisco IOS-XR platform that is mainly a ISP device.

From this blog, I found some interesting links. I didnt listen to the podcast though.

Awesome Connective Repo: Where I found this map about submarine cables (It rings the bell I saw it before) And this map about CDNs.

From this blog, I could read an interesting presentation about network performance using satellite services. I guess there are many blogs about the performance of Starlink but this was the first time I read something about it. I was surprised the results were not that good even with the latest version of Startlink (that I didnt know neither)

I checked out this blog note from Google about Falcon. To be honest, I dont really understand the “implementation”. Is it purely software? Does it interact with merchant Ethernet silicon ASICs? There is so much happening trying to get Ethernet similar to Infiniband that I am wonder how this fits outside Google infra. At the end of the day, Ethernet has been successful because everybody could use it for nearly anything. Just my opinion.