I am a bit obsess with productivity, trying to always produce more. This is another approach. He became Doctor via discipline but things started to fail when working as a junior Doctor. The idea is to move from discipline to joy. I find it difficult in general. I think I am not bad at discipline, but still got a pit paralysing when hitting the unknown, and splitting things in small chunks is helping me.
Energise:
Find/add the concept of “play” in what you do (sense of adventure and take things less seriously)
Find empowerment, choose how you work on things (most of the times you can’t choose on what your work…. so you know) You can choose the process and your mindset.
Surround yourself of the correct people that lift you up instead of suck up the energy. Think of giving help, it is not always getting.
Unblock:
This part focus in procrastination and the most important for me
Clarify: Think about what is blocking your mood, ask “why” you want to do something, then figure out “how”. For the “what” use NICE (near-term, input-based, controllable and energising) and then set the “when” you do things. Add “clues” to trigger the action (create an habit)
Fear: understand what you are afraid of? will that matter in 10 minutes, 10 weeks, 10 years? And very important, most people are not thinking about your, they are too busy worry about what the others are thinking about them.
Start: This is for me the most important part. Find the friction, what it is stopping you to start, how can you remove them? You dont have to get everything figure out, so “just do it” (do something). Be kind to yourself, analyze what you do, correct and move on.
Sustain
This is part is about burn-out. It can over-work, it can be boredom. Focus in doing less but better. Focus on “Fuck yeah or not” to decide things. Control the distractions (noise, youtube, etc) and take breaks where you do nothing.
You need to learn to recover and disconnect: have a hobby, try to be in nature (walk to a park)
Discover misalignments: you spend time on goals that dont align with your values.
This is a book from investor Peter Thiel. I am focus in the investment side and not in the political, that reminds me to Musk….
The idea from “zero to one” is to create something that doesnt exist, that is where the big profit can be created.
Mentions that is very important the personality of the founder and the size of the management team (the smaller the better) and how much is the founder paid itself (should be less than others to set the mood). All these is important for the success of the startup.
He talks about Paypal and the merge with X.com (Musk). He want monopolies because competition reduce profits, this was funny because from “Chip Wars” it was the contrary, you want competition and mass market to be profitable. Although the idea of his monopoly is a progressing one, not one that stops the development. The idea is “nice” but I dont see it happening… there are humans involved…
He talks about the PayPal mafia, that most of them afterwards found multibillion companies.
It is an interesting read about the the evolution of microchips development in a non-technical point of view.
Everything started with investment from the USA military. Transistors were meant to replace vacuum tubes, and make systems more reliable and efficient. Then the companies wanted to scale because the military didnt produce enough profit and then the competition to produce transistor at scale really created the boom.
During the cold ward, URSS only copied chips and was always behind, they didnt have the market to develop the technology and always stated behind.
The move of manufacturing to south-east Asia was meant to empower allies and stop China. The success of Taiwan (TSMC) and South Korea (memory) is based on the constant government support in providing low interest credits via banks and control of the currency to make exports cheap.
I didnt know that initially the manufacturing was focus in memory. Intel started to lose the battle and then moved to CPU… and was a huge success until the lost the mobile market (said no to iphone)
I was surprised of the role of Sony at the beginning and then the fall of Japan, and havent recovered. And how the sanctions crashed Huawy but now it is evolving and pushing China to be more reliant in its own product, although USA still have the last voice because in any part of the production change has a grip (laser technology for UEV, tools for designing chips, influence in Taiwan/TSMC)
And the history of ASML and the development of UEV. Super interesting.
In general, really good read to understand the political-economical of microprocessors in our world
Dont break the routine (x): consistency beats intensivity
Defaulting to “No”: You need to focus in what matters. I have a very strict routine: climbing, german, dancing, baking, reading, studying. And I should add dating. I have enough if I add work and sleep. I would like to do more things, yes, but dont have time. Although I feel sometimes a slave of my own routine. No means to focus in what matters to you. Time is finite.
If you think of somebody, reach out to them: I dont have many friends, but they are really good one, I am happy for that. I need to make the effort to reach out more often.
Quit drugs and alcohol: This is a no brainer. Doesnt need much explanation.
Buy your time back: I agree with this, and it is difficult. Learn how to spend your money. For me, it is repairing my bike. I realize that just being able to repair a puncture and changing the breaks it is enough. More means, buying a lot of kit and still need time. And likely more things.
Have fewer opinions
Stop complaining: Amen. If you do, have a plan of action.
Optionality Addiction: We have to many options, we dont focus. When you do, you have grief from the options lost. Committing means to do less. Freedom is the enemy of focus. Deliberate practice = tolerance boredom. Discipline = remove choice
LR4: I am not an expert in optical networking but it is an interesting read.
MPTE: Interesting the author point that even in a RNG (Random Network Graph – AWS) still is traffic engineering needed MPTE (Multipath Traffic Engineering). He makes his point based on AI environments.
Chocolate Filling (keep it in fridge so it is easy to chop)
Unsalted butter, softened: 45 g
White chocolate – 100g
Milk chocolate – 100g
Dark chocolate – 100g
Brushing & Topping:
Milk + softened butter – for brushing before baking
Chopped chocolate – reserve some from the filling for topping
Process:
Mix milk, yeast, sugar.
Add eggs, pinch salt, melted butter and mix
Add flour shifted and mix
Then manually knead until smooth
Rest for 1h
Chop the chocolate
Dough should be double, diflate and roll it in a surface with flour. Make a big rectangle
Brush melted butter apart from one side so you can seal it when rolling
Trim edges if needed
Spread chocolate evenly across the whole surface. Press with the rolling pin
Roll the dough tightly up. Be careful
Pinch the edges and the seal.
Cut the log in half, lengthwise.
Braid the two halves.
Move the log to the loaf pan (with baking paper)
Rest for 20 minutes
Brush it with the butter-milk mix.
Bake at 180°C for 30–35 minutes, or until beautifully golden brown. Dont use the grill
Brush it again with butter-milk so it shines.
This is my result:
It doesnt look as nice as in the video but it was tasty. And it is not actually difficult. I made mistake as using the grill and I think I should have tighted the roll better and use the rolling pin to press the choco chips. I was surprised how easy was to knead it.
This book is about the value of rethinking (mental flexibility). This is a path to learning more and living with fewer regrets. This can be hard because it means abandom parts of oneself. The process is based always in the scientific method, formulating a theory and trying to find counterexamples.
The process of learning is not to affirm our beliefs, it is to evolve them. And being (too) smart can be counterproducent because narrows our mind to what we dont know. I liked the example from the iphone, when Steve jobs was adamant of the product because would kill the ipod money making, or stubbornness from Blackberry to evolve. So in this case, you need a balance between confidence and competence.
These are the mental models that need to rethink
Cult leader: I am always right!
Politician: We are right, they are wrong!
Preacher: I am right
Prosecutor: You are wrong.
I like the references to the “impostor syndrome”. It always keeps you on your toes and push you to grow because you never know it all.
Who you are should be about your values and not your believes. That way we can be flexible. For example, when making a forecast, make a list of conditions that should hold true and other list of conditions that should change it. This makes you honest and detached from an opinion/believe.
This can produces discussions. But need to see it in the positive way, intellectual chemistry is enjoying disagreements. We learn more from people who challenge our thought process than those who affirm it (focus is in your thought process and not in you as a persona, that is a totally different matter). These conversation are focused in the “how” instead of “why”. Giving too many facts about are reasoning make the other part defensive (coming too strong). As well, we need to be open to change ours, find common ground. This is from the debate between IBM computer and Harish. As well you dont change people’s opinions, they do themselves, and based on “You can lead a horse to water, but you can’t make it drink.” So if the other person says there is nothing that would change its mind, then…. So it is around asking questions, be invested, like a dance and not a war (it is called motivational interviewing: examples like vaccines ) To be honest, I struggle to ask questions to people, like I am not curious (or it is a way to hide… but when people ask me, I talk….)
There is a part about applying to a jobs and using an example where the person didnt hide its shortcomings, just ack-ed in the paper and then went with the reason why it should be hired. I need to apply that to myself.
Some topics are difficult. Just acknowledge the complexity, show the range of perspectives: gun control, inmigration
Regarding learning, the best way is to teach it. And the active learning (more than the lecture type). Repeating is good = “jack of rough drafts, master of crafts”. So education should be more than gathering information, but develop the habits to revise our drafts and keep learning. I think this matches the IT culture of continues intergration/delivery, error soon and often, etc This creates the culture of growth and rethinking instead of sticking to “tradition/best practices” (examples from the NASA accidents) This takes too humility (improve better than proving oneself)
Another good one: Sunk cost (difficult to turn around when you are deep in something, and you can’t stop carry on with it) / Grit. Gritty mountaineers are more likely to die on expeditions. “There is a fine line between heroic persistence and foolish stubbornness. Sometimes the best kind of grit is gritting your teeth and turning around” I think the whole book goes around to when to do exactly that.
In general, explore (action) and then dig. And save time for a couple of times a year to do check-ups and rethink what you are doing.
The book finishes with the “take action” speech from Roosevelt in 1932. Accepts uncertainty, ack errors, learns from others and rethink plans
The book gives a different view from other sources, where they tell you to get grit, no plan B, etc etc. This proves there is no solution for all. So I take it as another tool.
This was another summer reading. Quite interesting because the author takes ageing as an illness and not as the last part of life.
I have come to realise that yes, we live longer, but not healthier… and I think there is an interest for that so you are hook to some medicines (always business). Personally, I dont see the point to be old and not being able to do anything for myself. Even worse, when your brain starts to fail… you dont even notice.
The first part is mainly the research and why the author says ageing is an illness. If shows how organism have evolved based on reproduction and repair. He explains it as “The Information theory of ageing” The DNA is digital but then the building process are analogical and that is where the information leaks and the problems starts. I can’t really explain all the biological details but it is an interesting theory.
How to live longer? He recommends many things, from fasting, eating less meat, exercise, cold/hot endeavours, avoid toxic materials and obviously there are pills you can take. To be honest all the recommendations look nothing special, I think they are sane, and is something good because everybody can do it.
The last part is the ethical / philosophical part. This is the hardest because that changes may things in our world. There will be more people (we need to produce more food! consumption – capitalism), people will work longer so less opportunities for young generations, politically we will be in a much more conservative (politically – super-rich want to stay…) world, how religion will handle people leave longer?, how people will handle to live so long?, will people want to die? It is interesting.
The author wants goverments to fund research in ageing because that can have more benefits and will solve problems like cancer or brain illnesses.
It is a bit psychological / philosophical about how to trade, live or just making decisions. Life is always changing and one must adapt and always keep learning/reading.
The main thema is the multidisciplinary so one can different mental models to discover a pattern. He gives a lot of stick about how badly is psychology taught in schools and universities (although he has only read a couple of books!) It mention the importance to have check-list (like pilots, surgeons, etc) and practice your mental models!
Their investment has been always long-term and based on compound-interest and rarely in Tech companies. I think their strategy is very different from Investment banks, Hedge Funds, etc. But I wonder if that model is going to carry on working in a market where those long-term investments are going to be still valid.
Anyway, I will not be rich or good investor (not my goal) but interesting the focus in keep learning/reading and having different models in your head to try to understand the world.
I have added several links about MRC before, but wanted to read the OpenAI paper for myself.
0 Abstract tail latency dominates performance of synchrounous pretraining jobs MRC (Multipath Reliable Connection) = new RDMA-based transport and a bit of UET, packet spraying, adaptive LB based on ECN, out-of-order memory placement, select ack and packet trimming to mitigate incast. RDMA: only support write and write-with-immediate multi-plane clos -> >100k in 2-tier, high switch radix static source routing SRv6, bypass failures
1 Intro solution needs: LB, incast, handle link and fabric failures gracefully disable dynamic routing!, only static paths with SRv6
2 Multi-plane topology co-design T0=TOR, 64x800G. Use 8x100G instead -> 8 planes of T0. TO connects to 256 NIC and 256 T1 (spines) -> total 131072 GPUs in 2-tier many advantages: latency, more nodes reachable in 1 hope, less cost and power (2-tier only), less impact link failure problems: survice link and nic port failures. load all network planes and LB all many paths!
2.1 MRC overview extends RoCEv2 + UET features (multi-path)
each packet has the RDMA virt address and remote key -> receiving nic can write at arrival, no matter order.
each packet has an entropy value EV (32 bits) Sender rotates the EV for each packet -> LB (packet spraying)
PFC disabled (it doesnt work with so many possible paths)
fast selective retransmissin = Selective-ACK.
under incast -> packet trimming = packet is not dropped, reaches destination, destination triggers NACk = retranmission Each EV corresponds to a specific path on a specific network plane
For each EV, MRC keeps a few bits of state about path health. ECN used but disabled in the last hop to the receiver -> ECN acts as LB signaling (different
MRC senders do not coordinate when choosing their EV sets)
When packet lost, EV path is stopped then MRC sends background probes to verify, if enough probes succedd -> EV resurrected
2.2 Static SR difficult to maintain large ECMP sets. MRC avoids failed paths, then dynamic routing re-routes -> changes ECMP -> disturb LB => disable dynamic routing How to map EVs to network paths? In the MRC nic, at QP startup a set of EVs are chosen, bits in each EV directly embed the path choice available at each hop in the network. SRv6: use microsegment ID (uSID), where destination ipv6 is 32b prefix followed by a seq of 16b uSIDs each corresponding to a specific switch along the path. Each swith left-shifts the uSID part of address. Switch has a static forwarding table, configured when installed, rarely changed. uN forwarding is done at line rate. MRC packets are ipv6 in ipv6 encapsulated. outer dst = srv6 path, inner dst = dst nic
2.3 Mapping EVs to SRv6 address with srv6, ev is not hashed by switches, but sill needs to be carried in data packets so received can echo it. SRv6 address can’t be echoed because it is erasedd by the shiffiting process during forwarding algo mapping between EV and its SRv6 address. *** COMPLEX need to review last part
2.4 Choosing working paths On QP startup, MRC must select a set of EVs, each mapping to a unique path on a specific plane. EVs are equally split across the planes, then randomly selects a subset of path within each plane. No necessary to use Clustermapper to set denylists for failed T0-T1 links. *We have to rum-up large training jobs slowly to avoid destabilizing the power grid. On the reverse path:
Reverse path: many QPs are unidirectiona at any instant. Keep small reserve EV set for control packets, with at least one EV per plane. Each RTT that the QP is acive inbound but generates no outbound traffic, the receiver send an EV prbe packet using a randomly chosen EV. If the probe is acked the reverse EV for that plane is set to the probe’s EV. If data traffic is sent, the reverse EV is updated from data SACKs instead of needing EV probes.
3 Operations With MRC, Most network failures dont require quick reactions from netops. Link-failures and flapping links between T0-T1 can be ignored!!! MRC spreads traffic across enough paths that when a link used by a QP flaps, only a very small number of packets per QP are lost, and the corresponding EV is removed. The missing pacet is selectvily retransm on a different path. MRC maps paths out when they drop, and only brings back when enough probes succedd over time -> less coordination with netops Can reboot T1 switch easily. NIC-T0 and T0 switches are more problematic -> After link drop, remapping all the EVs fro the many QPs in use is not instant. MRC uses a port state bitmap in SACK packets to notify remote QP endpoints that the port is down, so they also remap their EVs to avoid the failed plane. This takes a few seconds. CLustermapper agent running on all cluster nodes together probe every link in the network every ms!! This check NIC-T0 links and T0 switches. Agent send probes source-routed to the T0 and back to the same agent. Each agent also probes a subset of T1 switches, so all T0-T1 links are probed. ICMP probes are handled by CP, so low frequency. With SRv6, the switch treats a probe like another data traffic, by DP -> higher frequency.
4 Inter-plane Loading Keep all planes evenly loaded <- when MRC maps an EV out of its active set, it replaces it with one from the same plane -> aovid false incast at destination
5 Experiments flows: T0-local or cross-T1
5.1 Training Results
Link flaps almost no impact -> low priority at operations level
NIC-TO links does affect performance, but most events are transient and job recover full speed quickl. optical transceiver glitch in T0. NIC and Switch dont match the time when the port was down. Still job didnt crash and fully recover. This is rarer that individual link flaps.
T1 switch reboot: QP packet lost only at T1 failure. reboot took 2 minutes and no impact during reboot as all QP wer mapped out of the bad path. throughput largely unaffected.
NIC transceiver are SPOF, they flap, but rarely. A different optical design might avoid this issue.
5.2 Testbed results (MRC in different NICs: mellanox cx-8, amd pollara, broadcom thor ultra)
5.2.1 p2p communicatio performance – cluster B cx-8. t0-local and cross-t1 = aprox 770Gbps (96% theorical peak bw) t0-local 5.09 us and cross-t1 6.54 us (extra switch hop). High latency effect only short-message, negligible for large
5.2.2 MRC response to link down and flap events – Cluster B less 1 sec interruptions during failure detection . MRC detecs link down event and rebalances EV across remaining planes and updates link0state btmap in SACK packets to notify remote endpoint. T0-T1 failures have a substantially smaller impact than nic-t0 failures, because MRC sprays packets from each QP across a large number of paths.
5.2.3 MRC behavior with T0/T1 switch failured – cluster B T0 switch failure -> bw drops 100Gps. MRC failure handling is driven by end2end path availability rather than specific location or type of underlying fabric fault -> swithc-level failures manifest a predictable reduction of usable path diversity, preserve app progress and maintein stable throughpput T1 switch failure -> no degradation is observer. QP are sprayed across a large number or paths and sufficient alternative EVs remain available to sustain lost of capacity.
5.2.4 Robustness to path-level packet loss Introducing errors in a EV, caused it to get inactivated and activate another “spare” EV
5.2.5 Load balancing accross EVs Upon detection of congestion via ECN, MRC triggers traffic distribution to rebalance load across the available EVs. MRC can effectively rebalance traffic across EVs in repspomse to dynamic load changes.
5.2.6 NCCL collective execution at scale sendrecv benchmark: 92GBps in 42k GPUs
5.2.7 Comparison with RoCE MRC 1QP sparying across 256 paths achieves better performanc that RoCE with 16 QPs At 1% loss, RoCE is not usable. The same for MRC because the induced lost affected all planes. For all message sizes MRC outperforms RoCE but the differnce is greater in the regime that is bw-bound (larger message size)
At smaller message size, the collective is more latency-bound At bigger message size, the collective is more bandwidth-bound
5.2.8 Collateral Damange (Incast)
Configuring DCQCN properlyis very hard because it is traffic pattern specific to the point some hyperscalers have disabled it.
6 Related Work Load balancing effectiveness is defined by distribution granularity: per flow, per subflow and per packet. RocE2 uses per-flow ECMP. Per-packet load balancing (or packet spraying) has been proposed for CLOS networks to reduce amounf of state per (virtual) path -> use ECN or similar
Multipath TCP: must keep state per subflow. Used by Google Falcon to replace RoCE2 in Nic
MRC changes RoCE2 to enable packet-level load balancing, selective retrans and improve reliability in multi-plae networks: its target deployment is best-effort (lossuy) networks that support packet trimming.
MRC is similar to UltraEthernetTransport (to replace RoCE2)
7 Conclusion MRC is designed to LB a multi-plane network by spraying each QP across all planes and many paths in each plane, performing fine-grain active LB and routing around failures. Disabled dynamic routing, use SRv6 source routing with static routes in the switch
This is blog about MRC from a competitor. Really to see how other side sees it and shows the limitations and differences
