{"id":2177,"date":"2026-06-06T13:39:15","date_gmt":"2026-06-06T12:39:15","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=2177"},"modified":"2026-06-06T13:39:15","modified_gmt":"2026-06-06T12:39:15","slug":"friction-morning-routine-roce-meta-paper-aws-rng-slurm-rail-optimize-800vdc-phyllo-approach","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2026\/06\/06\/friction-morning-routine-roce-meta-paper-aws-rng-slurm-rail-optimize-800vdc-phyllo-approach\/","title":{"rendered":"Friction, Morning Routine, RoCe Meta Paper, AWS RNG, Slurm, Rail-Optimize, 800VDC, Phyllo, Approach"},"content":{"rendered":"\n

Manson AI<\/a>: You need friction<\/p>\n\n\n\n

AI Agent<\/a>: narrow focus: goal, proof, steps <\/p>\n\n\n\n

Bear Grylls’ Morning Routine<\/a>: Cold (never get used to), bared foot, strength training, 30 minutes<\/p>\n\n\n\n

RoCE networks for distributed AI training at scale<\/a>: I have managed to read the paper<\/a> ! Although in the AI word, two years is an eternity, I think it is still interesting.<\/p>\n\n\n\n

1) Network Topology<\/strong>: backend network only for GPUs (RDMA nics), non-blocking. Frontend network: data ingestion, checkpointing, logging.\n\nPod = AI zone\n leaf = RTSW, DAC cables, shallow buffer\n spine = CTSW, deep buffers. fiber between leaf-spine.\nSuperSpine = ATSW, oversubscribed, connect AI zones\n\nintra-node -> nvlink\nROCE: cpu offloading, ethernet (standard)\n\ncollective communication library serves as the sw abstraction between training workloads and the NIC\n                                 schedules verbs calls over QP (Queue Pairs)\n parallelism strategy determines collective: allreduce, allgather, alltoall\n choice logical topology:\n\n------------------\n\n2) Routing<\/strong>: work load. low entropy flows (few flows) -> ECMP bad (5-tuple udp: src\/dst ip, src\/dst port, protocol), burstiness, elephant flows\n--\n\n RTSW uplinks 1:2 under-subscribed! -> expensive (short-term)\n 1) QP scaling: use destination QP of Roce packet using the UDF capability in switch to increase entropy -> Enhanced ECMP -> short-term\n 2) Central TE controller -> long-term: CP real-time topology end-to-end cluster, \n                                        flow matrix (flow bps) + CSPF (constrained SPF)\n                                        write in switches dataplane\n                                     DP: TE overrides default BGP routing policy in leaf. Use Exact Match table.\n                           Not good with multiple link failures. Doesnt scale \n 3) Flowlet switching: try to improve 1 and 2. hw assistant schema. put packets in different ports in ECMP\n     out-of-order: move packets only after 1\/2 RTT\n     load-aware path assignment: better than TE\n\n------------------\n                     \n3) Transport<\/strong>: congestion management. Start with DCQCN. packet drops on ACK\/NACK can cause prolonged Local ACK timeout (LAT)\n--\n Tuning DCQCN not great (strict ECN -> minimize PFC (can lead to head-of-line blocking)\n\n 200G, we stayed with relaxed ECN marking, allowing for buffer build up in the CTSW, while keeping default DCQCN settings.\n 400G We proceeded without DCQCN. just PFC for flow control\n re-design collective library: two-stage copy\n\n------------------\n\n4) Operations<\/strong>:\n Change QoS priority of Clear to Send (CTS) messages. In RTSW ASIC, modify dsCP marking for ACK  messages\n Tuning VOQ in CTSW\n obeservability: OOS: out of seq.\n                 Link flaps\n                 Local ACK timeouts (LAT)\n                 PFC watchdog: catch any long-duration PFC pause (>200ms)\n                 buffer utilization RTSW\n                 reachibility (pings)\n                 constant latency monitoring loaded and unloaded (catch regressions)\n                 base lines!!!\n<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Perplexity: Hosting Qwen on Blackwel<\/a>:<\/p>\n\n\n\n
AWS RNG<\/a> – Random Graph Network: The paper<\/a> is totally out of my space, but the concept looks brutal. With an operations hat, how you troubleshot it? (ping, traceroute, link congestion, data flows patterns, etc)<\/p>\n\n\n\n
Slurm<\/a>: I like the “Slurm vs. Kubernetes”<\/p>\n\n\n\n
\n
Slurm Workload Manager (short for Simple Linux Utility for Resource Management) has become a cornerstone of large-scale computing. Originally created in the early 2000s to support large-scale high-performance computing (HPC) environments, Slurm is now widely recognized as the de facto scheduler for HPC clusters. Today, it orchestrates jobs across thousands of servers and GPUs<\/a> in some of the world\u2019s most advanced computing environments. <\/p>\n<\/blockquote>\n\n\n\n
Interview Question<\/a>: 512 GPU, non-blocking (full bisection) and 2xUFM! I really liked this. I think for once I understand the rail-optimize (fat-tree = leaf-spine). Just break one leaf-spine link, beautiful!!!<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
800VDC<\/a>: Next step in electrical infra in DC space.<\/p>\n\n\n\n
Phyllo by hand<\/a><\/p>\n\n\n\n
Approach woman<\/a>: curiosity and no performance. Practice. Be at peace with uncomfortable and akwardness. Rejection as learning<\/p>\n","protected":false},"excerpt":{"rendered":"
Manson AI: You need friction AI Agent: narrow focus: goal, proof, steps Bear Grylls’ Morning Routine: Cold (never get used to), bared foot, strength training, 30 minutes RoCE networks for distributed AI training at scale: I have managed to read the paper ! Although in the AI word, two years is an eternity, I think … Continue reading “Friction, Morning Routine, RoCe Meta Paper, AWS RNG, Slurm, Rail-Optimize, 800VDC, Phyllo, Approach”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,13,9,27,18,2],"tags":[],"class_list":["post-2177","post","type-post","status-publish","format-standard","hentry","category-automation","category-aws","category-cooking","category-kubernetes","category-maths","category-networks"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/2177","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=2177"}],"version-history":[{"count":11,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/2177\/revisions"}],"predecessor-version":[{"id":2198,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/2177\/revisions\/2198"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=2177"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=2177"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=2177"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}