Lightmatter: Based on this video, they are using photonics to connect chips, looks interesting, I remember Google has something with optical but for networking. But It is pretty clear this is not photonics computing.
Hacking USB cable: impressive, and expensive 🙂
Phil Stutz: Interesting conversation. But somehow, I am still looking for that thing that unlocks me…. can’t find it for the life of me….
- allreduce: collect elements from all nodes, apply a reduction operator(eg sum) then distribute reduction to all nodes
-allgather: collect elements from all nodes, and distribute the to all other nodes
- gpu: cpu for parallelization
- RDMA: RoCE2 GPU memory to GPU memory - origin in IB
- issues: flow collision, trafic polarization. low entropy!!! > dificult to ecmp => Dynamic LB
incast: many2one -> ECN + buffering (in spine!)
- use chassis!
With an operations hat on, dealing with chassis is expensive and no efficient. It kind of a vendor lock-in. AWS is all in pizza boxes and I remember one presentation in Cisco Live where the Cisco EVPN authority recommended pizza boxes.