Month: December 2023

TCP Port States – AI views – Prompt Engineering – 40s crisis – Unstoppable Motivation – 8 steps – Prisoners Dilema – Postgresql upgrade downtime

  • Interesting read about linux networking at TCP port level. It shows you how complicated can be but still amazing to understand Linux kernel.

  • Some views about AI:
    • 1) link1: Cory Doctorow about bubbles and AI. Models that run on commodity hardware will survive. He uses a 2×2 grid based on value (how much you will pay) and risk tolerance (how perfect the product needs to be)
    • 2) link2: Bill Gates views for AI. It is the word for 2023. And future for 2024. Still I think in link1, AI has to be democratic and easily available. I doubt most developing countries can afford NVDIA H200 in Azure…. I agree nuclear (fusion) energy is the only way forwards in the Western world until we have viable fision energy.
    • 3) link3: Prompt Engineering

  • Unstoppable motivation: This is an unexpected pearl. Intrinsic motivation is always preferred (more effective and energized) than extrinsic motivation (external elements: money, prizes, people opinion). But not always is so clearly binary. Extrinsic motivations can help too:
    • “Introjected Motivation. “I’m doing this because I’ll feel guilty or bad about myself if I don’t.” People who highly rated this statement have high introjected motivation.”
    • “Identified Motivation. “I’m doing this because I truly value the goal it’s helping me work towards.” People who highly rated this statement have high identified motivation.“
  • Because having intrinsic and extrinsic will give you more tools to move forwards when one of them weakens.
  • “the only type of extrinsic motivation that corresponded with greater happiness was identified motivation. In other words, it was the hikers who motivated themselves by aligning their actions with what they truly valued and who not only completed the trail—but also felt happiest at the end of it“
  • And the most important part of the article: How to get more intrinsic and identified motivation in your life:
    • Figure out what really matters to you: For me this is the most complex and scary part.
      • Long Term: The eulogy method
        • Medium Term: How will be your life in 5 years time if you follow Current path, alternative path and radical path? money, social obligations, and what people would think, are irrelevant. No excuses
        • Short Term: Wheel of life: measure your health, work and relationship.
    • Definitely I need to do the medium and short term. That would clarify many things…
  • 8 steps: i have read about rules but this keeps some basic into perspective:
    • Take care of yourself: shower, dress, etc
    • Order your room, kitchen, etc: your kingdom = yourself
    • Go outside – socialize
    • Sweat: work out
    • Money: control your economy.
    • Remove dependencies: sugar, diet attention (emails, youtube)
    • Strategy – create your action plan.
    • Execute: dont overthink
  • 40s crisis: I think we “overestimate” our cultural background, we think our lives are going the same phases as our grandparents or later. Quite wrong. This is a new phase and “recent” (half XX century). We are at the peak and we only see ahead the downwards of life. Likely our grandparents were already at 40. We have the social life cycle. That creates age-anxiety if we dont meet that cycle. Divorce increases along the years in XX century to reorient live. Min 31:25 is a pearl: We dont begin to live until we begin to die. And the real American dream (not based on materialism / radical capitalism) and moving to happiness in a hurry (consuming) created that crisis because you couldn’t achieve that American dream.
  • Application of Game Theory: Prisioners Dilema: Simplest win, tit for tat:
    • – nice (vs nasty): you dont defect first
    • – forgiving: you can retaliate but forgives
    • – retaliatory: strike back immediately, dont be a push over.
    • – clear: eye for eye
  • Interesting is that a small cluster of tip for taps can take over a population of nasty strategies. Most life is not zero-sum, try to find win-win.
  • PostgreSQL upgrade without downtime: Slides are ok, but would be nice to get the video for more details.

Is This Anything?

I never connected with the serie Seinfeld (never tried to watch it neither), I knew about it, I knew people who were superfans and took it as nearly religion, but still I never had the curiosity. I remember many years ago that when the show finished, it was a big deal, people launched parties to celebrate the long success of the serie.

One day I read about the work behind being a stand-up comedian, and they put as example this book I have just read. The point, the humor/jokes dont come out as art of magic, it is a constant job. The book is about “jokes” since he started in 70s until 2010s so that shows you that there is no magic bullet.

I like some jokes, they made laugh a lot and there were many I didnt get. I guess it is not the same reading a funny line that watching it.

The best ones I remember are the one about security guards and terrorists in airports, Wrestling referees, Fakebook and dual AC in cars, to name a few.

And another point that I realized that is very important and I skip, is the need of humor/laugh. Sometime I think that all is about Stoicism, Buddhism and other ideas but I noticed that missed something. A good laugh, with good friends, and you see the world in a different way.

Hope I dont forget it

How To Learn

From Andrew Huberman. Based on these two videos: 1 and 2

Get Alert: Deep Breathing

Get Focus: Focus in a point for 30s

Time Limit: 90 minutes. Batching.

Gap Effects: breaks of 10 sec doing nothing

NSDR: Non-Sleep Deep Rest: napping or meditation.

Black Spartacus

I just finished this book. Very interesting life from Toussaint Louverture from a slave to being the first to declare the end of slavery in the Caribbean and triggering the independence of Haiti (he died before that). I think he wouldn’t be proud of what happened to his country after all his efforts though. Haiti reminds me of this movie.

I didnt know Napoleon wanted to establish again slavery in Haiti once Toussaint declared the constitution. I always thought that after the French Revolution, freedom would be given for granted. You know all the “liberte, egalite, fraternite”… it seems only applied to France (and whites). Something we dont learn in Europe.

As well, the Haiti flag is like the French one without the white part. They removed it because they were angry with the French and whites after France tried to invade Haiti.

As well, for being the first colony to declare the end of slavery or trying independence, it is not as well know as the Cuban revolution or Simon Bolivar. Toussaint was ahead of his time and showed the path for the next generation. He declared the Haiti constitution in 1801. The American Civil War started on 1861.

It is interesting his Republican and Catholic credos. His constant show of discipline and how he became a great general winning and fighting battles. And he managed to take over the current Dominic Republic. And deal with the main power of the time: England, Spain, France and USA.

But again, I think power corrupts any soul, and the longer you have it the worse. The last years of Toussaint shows social issues. It is remarkable his view of a equal society, no matter the color of skin. Education and religion for all. Same opportunities for all. But it hit me when in the book mention that most of his generals owned big plantations but the normal black worked had no access to property or not was promoted. It seems workers were paid a 1/4 of the production that is not bad but at the end, you want the chance to be your own boss. He wanted to increase the agriculture production and that made to create laws that look more as dictatorship than anything else.

It is a pitty that he didnt have a (declared) successor that would be at his level.

BIND performance – LACP Troubleshooting – Chiplets – AI/HPC Networking – Spray ML/AI workloads

BIND: Interesting links about BIND performance and the lab setup. DNS is the typical technology that looks straightforward but as soon as you dig a bit, it is a world itself

LACP: Interesting blog about troubleshooting details. As above, this is the typical tech that you give for granted that works but then, you need to really understand how it works to troubleshoot it. So I learned a bit (although the blog is “old”)

Chiplets: Very good blog. Explaining the origin of getting to chiplets. Interesting evolution and good touch to mention the network industry, and not just CPU/GPU.

As the process node shrank, manufacturing became more complex and expensive, leading to a higher cost per square millimeter of silicon. Die cost does not scale linearly with die area. The cost of the die more than doubles with doubling the die area due to reduced yields (number of good dies in a wafer).

Instead of packing more cores inside a large die, it may be more economical to develop medium-sized CPU cores and connect them inside the package to get higher core density at the package level. These packages with more than one logic die inside are called multi-chip modules (MCMs). The dies inside the multi-chip modules are often referred to as chiplets.

AI/HPC Networking: Nice summery about AI vs HPC, and what each hyperscaler and vendors are doing. For me is quite interesting how to get proper loadbalacing of flows like AWS SDR. This should be an actual standard by any network vendor or software to aim to that goal. I guess it is not easy.

High performance requirements can create a vendor lock-in. Doesn’t matter if it is IB or Ethernet. So pick your evil.

Spray ML/AI worloads: Based on the above regarding the loadbalacing, this is an interesting article about how to generate loadbalancing in ML workloads when it is based in just one elephant flow. So you need Adaptive routing in your fabric/switches, NICs that support it and support from your code/library.

AWS Intent-Driven 2023- Groq – Graviton4 -Liquid Cooling – Petals – Google – Crawler – VAX – dmesg

AWS Reinvent Intent-Driven Network Infra: Interesting video about Intent-driven networking in AWS. This is the paper he shows in the presentation. Same note as last year, leaf-spine, pizza boxes, all home made. The development of the SIDR as the control plane for scale. And somehow the talk about UltraCluster for AI (20k+ GPU). Maybe that is related to this collaboration NVIDIA-AWS. Interesting that there is no mention to QoS, he said no oversubscription. In general, everything is high level, and done in-house, and very likely they facing problems that very few companies in the world are facing. Still would be nice to open all those techs (like Google has done – but never for network infra). As well, I think he hits the nail on the head how he defines himself from Network Engineer to Technologist, as at the end of the day, you touch all topics.

AWS backbone: No chassis, all pizza boxes

Graviton4: More ARM chips in cloud-scale

Groq: Didnt know this “GPU” alternative. Interesting numbers. Let’s see if somebody buys it.

Petals: Run LLMs bittorrent style!

Google view after 18 years: Very nice read about the culture shift in the company, from do not evil, to make lots of many at any cost.

GTP-Crawler: Negative thing, you need the pay version of chatgpt. I wonder, If I crawke cisco, juniper and arista, what would be nearly all network knowledge in the planet? If that crawler can get ALL that date.

Linux/VAX porting: Something that I want to keep (ATP).

dmesg -T: How many times (in even more years!!!!) I wondered how to make those timestamp to something I could compare with then debugging.

Team Topologies

I am reading this book as part of a book club at work.

The summary for the first part (chapters 1-3)

1 The problem with organization charts:

The software tends to match the organization charts (top-down) and that is not a good approach for software development because that doesnt cover (current) good software practices (devops, agile, lean, etc). This is based on Conway’s law: “Organizations which design systems . . . are constrained to produce designs which are copies of the communication structures of these organizations.”

So instead of using org charts, move to a “teams” organization approach that authors call “team topologies”: adaptive model for technology organization design that allows businesses to achieve speed and stability. Focuses on how to set up dynamic team structures and interaction modes that can help teams adapt quickly to new conditions, and achieve fast and safe software delivery.

Cognitive load can be a powerful tool for deciding on team size, assigning responsibilities, and establishing boundaries with other teams.

Dan Pink’s three elements of intrinsic motivation: autonomy (quashed by constant juggling of requests and priorities from multiple teams), mastery (“jack of all trades, master of none”), and purpose (too many domains of responsibility)

Personally, I think the org chart still has a point from HR perspective, at least I want to know who is who.And disagree when they say “there is a strong focus on the more immediate automation and tooling adoption, while cultural and organizational changes are haphazardly addressed”. People dont change tools that easily. And company culture hardly is checked out.

2 Conway’s Law and Why It Matters

The idea to become a high-performing organization is to reverse Conway law as a team-first software architecture. Then you need things team sized. The size needs to maximizes people’s ability to work with it. Important to define “team interfaces” to set expectations around what kind of work requires strong collaboration and what doesn’t (some type of communications are not needed). Tools selection is important, one doesn’t fit all. Unnecessary re-org (without any proper justification)/cut headcounts are bad (obviously).

3- Team-First Thinking

Follow: team-first architecture. Examples:

-US Army have adopted the team as the fundamental unit of operation. Based on scaling by Dunbar (5-7 people max)
-google: teams matter more than individuals.
-aws: team of size to feed with 2 pizzas)


Maximize trust between people on a team, and that means limiting the number of team members. Adding new people to a team doesn’t immediately increase its capacity (Fred Brooks – The Mythical Man-Month)

The best approach to team lifespans is to keep the team stable

Every part of the software system needs to be owned by exactly one team: no shared ownership

You need in the team:
– Team-First Mindset
– Diversity
– Rewards teams, no individuals
– Restrict Team Responsibilities to Match Team Cognitive Load more space for germane cognitive load (which is where the “value add” thinking lies).
team needs the space to continuously try to reduce the amount of intrinsic and extraneous load

Measure the Cognitive Load Using Relative Domain Complexity: mastering a single domain -> mission is clear, less context switching.

Limit the Number and Type of Domains per Team: simple, complicated, complex. Complex only one team,

Design the system and its software boundaries to fit the available cognitive load within delivery teams.

To increase the size of a software subsystem or domain for which a team is responsible:

  • team-first work env
  • minimize distractions
  • goals and outcomes (good) vs how (bad)
  • dev-exp

Teams need a team API and well-defined team interactions (aws: everything is an API): code, version, wiki/doc, principles, communication/information

As well, provide time, space, and money to enable inter teams collaboration

Explicitly Design the Physical and Virtual Environments to Help Team Interactions: I think the physical site is kind of clear (there are good notes) but for the virtual env they just mention group chats.

physical: office layout, whiteboards, standing desks?, noise, clear desk (lockers)
(squads -> tribes)

Summary: Limit Teams’ Cognitive Load and Facilitate Team Interactions to Go Faster

===

part2

In Part I, we saw the strong pull that Conway’s law exercises on system architecture by mirroring team structures and communication paths in the final product design. We also highlighted that efficient software delivery requires a team-first approach that relies on long-lived autonomous teams achieving fast flow. Part II will focus on how we put these two ideas together in a way that maximizes flow yet respects the cognitive limits of teams.

CHAPTER 4: Static team topologies

  • Instead of structuring teams according to technical know-how or activities, organize teams according to business domain areas.

Summary

  • Ad hoc or constantly changing team design slows down software delivery.
  • There is no single definitive team topology but several inadequate topologies for any one organization.
  • Technical and cultural maturity, org scale, and engineering discipline are critical aspects when considering which topology to adopt.
  • In particular, the feature-team/product-team pattern is powerful but only works with a supportive surrounding environment.
  • Splitting a team’s responsibilities can break down silos and empower other teams.

Notes:

  • team anti-paterns
    –Organizations must design teams intentionally by asking these questions: Given our skills, constraints, cultural and engineering maturity, desired software architecture, and business goals, which team topology will help us deliver results faster and safer? How can we reduce or avoid handovers between teams in the main flow of change? Where should the boundaries be in the software system in order to preserve system viability and encourage rapid flow? How can our teams align to that?
  • design for flow of change: Spotify: journey in progress, not a journey completed.
  • Shape team intercommunication to enable flow and sensing:
    — Organizations that value information feedback from live (production) systems can not only improve their software more rapidly but also develop a heightened responsiveness to customers and users.
  • devops and the devops topologies
    –A key contribution of DevOps was to raise awareness of the problems lingering in how teams interacted (or not) across the delivery chain, causing delays, rework, failures, and a lack of understanding and empathy toward other teams.
  • devops topologies
    –The DevOps Topologies reflect two key ideas: (1) There is no one-size-fits-all approach to structuring teams for DevOps success. The suitability and effectiveness of any given topology depends on the organization’s context. (2) There are several topologies known to be detrimental (anti-patterns) to DevOps success, as they overlook or go against core tenets of DevOps. In short, there is no “right” topology, but several “bad” topologies for any one organization.
  • successful team patterns
    –The success of different types of teams does not depend solely on team members’s skills and experience; it also depends on (perhaps most importantly) the surrounding environment, teams, and interactions.
  • Feature Teams Require High-Engineering Maturity and Trust
    –The feature team typically needs to touch multiple codebases, which might be owned by different component teams.
    –someone still had to keep oversight of the system as a whole and ensure subsystems integrated and interacted according to the desired user experience, performance, and reliability. –> create specific roles: system architects, system owners, or integration leads.
  • Product Teams Need a Support System
    –The key for the team to remain autonomous is for external dependencies to be non-blocking, meaning that new features don’t sit idle, waiting for something to happen beyond the control of the team –> Non-blocking dependencies often take the form of self-service capabilities
  • Cloud Teams Don’t Create Application Infrastructure
    — cloud team providing high-quality self-services that match both the needs of product teams and the need for adequate risk and compliance management.

-SRE Makes Sense at Scale
–The SRE model sets up a healthy and productive interaction between the development and SRE teams by using service-level objectives (SLOs) and error budgets to balance the speed of new features with whatever work is needed to make the software reliable.
–SRE is a dynamic balance between a commitment to operability from the application-development team and expertise from the SRE team. Without a high degree of engineering discipline and commitment from management, this fine balance in SRE can easily degrade into a traditional “us and them” silo that leads to repeated service outages and mistrust between teams.

-Considerations When Choosing a Topology

-Technical and Cultural Maturity

-Quadrant: Organization Size / Software Scale vs Engineering Maturity

-Splitting Responsibilities to Break Down Silos
–highlight the importance of thinking about teams’ capabilities (or lack thereof) and how that causes dependencies between teams.

-Dependencies and Wait Times between Teams
–three different categories of dependency: knowledge, task, and resource dependencies.
–Detect and track interdependencies.

-Detect and track interdependencies.

Summary: Adopt and Evolve Team Topologies that Match Your Current Context

CHAPTER 5: The Four Fundamental Team Topologies

  • The architecture of the system gets cemented in the forms of the teams that develop it
  • Four fundamental team topologies:
    –Stream-aligned team
    –Enabling team
    –Complicated-subsystem team
    –Platform team

There is no “Ops” team or “support” team in the fundamental topologies,

Summary

  • The four fundamental team topologies simplify modern software team interactions.
  • Mapping common industry team types to the fundamental topologies sets up organizations for success, removing gray areas of ownership and overloaded/underloaded teams.
  • The main topology is (business) stream-aligned; all other topologies support this type.
  • The other topologies are enabling, complicated-subsystems, and platform.
  • The topologies are often “fractal” (self-similar) at large scale: teams of teams.

Notes

-Stream-Aligned Teams
–A “stream” is the continuous flow of work aligned to a business domain or organizational capability. Continuous flow requires clarity of purpose and responsibility so that multiple teams can coexist, each with their own flow of work.
–The stream-aligned team is the primary team type in an organization, and the purpose of the other fundamental team topologies is to reduce the burden on the stream-aligned teams
“you build it, you run it” popularized by Werner Vogels, CTO of Amazon

-Capabilities within a Stream-Aligned Team
–This might mean having a mix of generalists and a few specialists.

-Why Stream-Aligned Team, Not “Product” or “Feature” Team?
–a stream should flow unimpeded.

-Expected Behaviors in an effective stream-aligned team
— several features:
–roduce a steady flow of feature delivery, uses an experimental approach to product evolution, expecting to constantly learn and adapt., minimal (ideally zero) hand-offs of work to other teams., evaluated on the sustainable flow of change it produces, must have time and space to address code quality changes (tech debt), proactively and regularly reaches out to the supporting fundamental-topologies teams (complicated subsystem, enabling, and platform).feel they have achieved or are in the path to achieving “autonomy, mastery, and purpose

-Enabling Teams
–An enabling team is composed of specialists in a given technical (or product) domain, and they help steam-aligned team with improving their capabilties (read, learn, practice new skills). It is a “technical consulting team” – gives guidance, not execution.

-Expected Behaviors
— proactivively seeks to understand needs of stream team, ahead of the tech curve, messenger of good/bad news, might act as a proxy for difficult services, promotes learning across teams
— Stream-aligned teams should expect to work with enabling teams only for short periods of time (weeks or months) in order to increase their capabilities around a new technology, concept, or approach.

-Enabling team vs Communities of Practice (CoP)
— Enabling teams and CoP can co-exist because they have slightly different purposes and dynamics: an enabling team is a small, long-lived group of specialists focused on building awareness and capability for a single team (or a small number of teams) at any one point in time, whereas a CoP usually seeks to have more widespread effects,

-Complicated-Subsystems Teams
–A complicated-subsystem team is responsible for building and maintaining a part of the system that depends heavily on specialist knowledge, to the extent that most team members must be specialists in that area of knowledge in order to understand and make changes to the subsystem.
–complicated-subsystem team is created only when a subsystem needs mostly specialized knowledge. The decision is driven by team cognitive load, not by a perceived opportunity to share the component.
–we expect to have only a few complicated-subsystem teams in a Team Topologies–driven organization

  • Expected behaviors
    –off-load work from stream-aligned teams on particularly complicated subsystems that need to be developed by a group of specialists.

-Platform teams
–enable stream-aligned teams to deliver work with substantial autonomy.
–The platform team’s knowledge is best made available via self-service capabilities via a web portal and/or programmable API — Ease of use.
–the platform can provide different levels of service.

-Expected behavior
–strong collaboration with stream-aligned teams to understand their needs.,fast prototyping techniques and involves stream-aligned team members for fast feedback

-Compose the Platform from Groups of Other Fundamental Teams
–ested or “fractal” teams within the platform—what we like to call inner topologies.

-Avoid Team Silos in the Flow of Change

-A Good Platform Is “Just Big Enough”

  • aim for a thinnest viable platform (TVP) and avoid letting the platform dominate the discourse.
  • Cognitive Load Reduction and Accelerated Product Development: The most important part of the platform is that it is built for developers.”

-Compelling, Consistent, Well-Chosen Constraints: Dev Experience (User experience UX)

-Built On an Underlying Platform (platform layers)

-Manage as a Live Product or Service: using software-product-management techniques

-Convert Common Team Types to the Fundamental Team Topologies

-Move to Mostly Stream-Aligned Teams for Longevity and Flexibility

-Infrastructure Teams to Platform Teams

-Component Teams to Platform or Other Team Types

-Tooling Teams to Enabling Teams or Part of the Platform

-Converting Support Teams: 1) support teams aligned to the stream of changes, and (2) dynamic cross-team activity to resolve live service incidents.

-Converting Architecture and Architects
–The most effective pattern for an architecture team is as a part-time enabling team (if one is needed at all)
–to help them achieve better outcomes and provide them the tools and technologies that will enable these outcomes.”

-Summary: Use Loosely Coupled, Modular Groups of Four Specific Team Types
–stream aligned, enabling, complicated subsystem, and platform.

CHAPTER 6

Summary

  • Choose software boundaries using a team-first approach.
  • Beware of hidden monoliths and coupling in the software-delivery chain.
  • Use software boundaries defined by business-domain bounded contexts.
  • Consider alternative software boundaries when necessary and suitable.

Notes