An Introduction to Matter

To start, what is matter? Matter is made up of molecules, and molecules are assemblages of atoms, the building blocks of matter. For example, water is an H₂O molecule: 2 hydrogen atoms and 1 oxygen atom.

An atom is itself composed of a nucleus and electrons, which carry a negative charge and orbit around it.

The nucleus contains two types of particles:

• Protons, which carry a positive charge, naturally repel each other.

• And neutrons, which carry no electric charge and act as a kind of “glue,” helping to keep the nucleus stable.

The attraction between electrons (−) and protons (+) keeps the whole thing in a stable state.

On the other hand, too few or too many neutrons relative to the protons, and the nucleus becomes unstable. It will eventually decay by emitting energy. This is the principle of radioactivity. Carbon-14, for example, is slightly unstable. It decays slowly and predictably. This predictability allows it to be used as a clock to date ancient elements.

One might think that when touching a solid object, like a table, for instance, what gives the table its solidity is that it is “filled” with matter, preventing our finger from passing through. Yet, if we took the nucleus and enlarged it into a marble and placed that marble on a football pitch, the electrons would be orbiting at the level of the stands.

An atom is therefore almost entirely empty. Solid matter is almost nothing, and what gives this impression of solidity are forces between atoms called electromagnetic forces.

The Fundamental Forces in the Universe

The universe is made up of 4 and only 4 fundamental forces:

• Gravity: Attracts everything with mass toward everything else with mass.

• The strong nuclear force: It glues protons and neutrons together inside the nucleus.

• The weak nuclear force: Responsible for certain radioactive decays. It is what allows a neutron to transform into a proton (or vice versa).

• And the electromagnetic force.

If we focus on this last one, it is the one that:

• Attracts opposite charges

• And repels identical charges.

Unlike the two nuclear forces, which only act inside the nucleus, the electromagnetic force has an infinite range. That is why it is the one that governs interactions between atoms at our scale.

It is therefore the electromagnetic force that creates the illusion of solidity. When we touch a table, it is the electrons in our hand and those in the table that repel each other. We never truly touch anything.

Light: Wave and Particle

Let’s now talk about light. So, what is light? It is an electromagnetic wave, a disturbance of the electric and magnetic fields that propagates through space.

Light is a spectrum. Indeed, so-called visible light, the light our eyes can perceive, is only a tiny portion of what exists. The full spectrum is called the electromagnetic spectrum:

Radio wave → Microwave → Infrared → Visible light → UV → X-rays → Gamma rays

When a radio picks up radio waves, it is therefore picking up light, invisible due to its frequency. Indeed, what varies across the electromagnetic spectrum is the frequency of the wave, and therefore its energy.

But light hides a surprise: it is also a particle.

NOTE: A particle can be summarized as follows: an indivisible packet of energy.

We know it is a particle thanks to Einstein in 1905 (for which he received his only Nobel Prize, not for relativity). When a light bulb emits light, it emits specific particles called photons.

When we vary the intensity of that light bulb, one might assume it is the energy of the photon that varies, but that is not the case. The energy of each photon is fixed by its frequency. The higher the frequency of a photon, the more energetic each photon is. That is why, for example, UV rays burn the skin.

What makes a light bulb emit more light is the increase in electric voltage, which therefore produces more photons. It is the quantity of photons that makes a light bulb shine more or less.

In flight, the photon behaves like a wave: it propagates, it oscillates, and it can interfere with other photons. But when it comes into contact with matter, it behaves like a particle: it interacts in one single hit, in one single place.

When a photon collides with matter, it can either be:

• Absorbed: The photon ceases to exist. Its energy is transferred to an atom, which moves to a higher energy level. This is what an eye does: it absorbs the photon and converts it into an electrical signal.

• Reflected: Technically, this is not a true reflection because it is not the same photon that leaves. The atom absorbs the photon and then re-emits a photon of the same energy in a different direction.

NOTE: What determines whether a photon is absorbed or reflected depends on the energy levels of the electrons in the atoms of the surface. If the photon’s frequency matches an available energy level, the atom absorbs it. Otherwise, the photon is re-emitted. That is why glass is transparent, why the retina absorbs light, and why a mirror reflects almost everything.

The Young’s Double-Slit Experiment

We have seen that light is a wave. But how do we know this? This is where Young’s double-slit experiment comes in, and it is this very experiment that will lay the foundations of quantum physics.

Young’s experiment, carried out for the first time in 1801, is as follows:

• A laser projects photons (light)

• A wall with two small slits, A and B

• A screen behind to detect where the photons land

If light were a “packet” of something, we would see the following result:

Yet, the result of Young’s double-slit experiment is as follows:

We obtain what is called an interference pattern. The wave passes through both slits simultaneously, splits into two, and these two waves meet on the other side.

When two waves meet, they add up or cancel out depending on their respective phase:

• Two crests meeting → they add up → bright zone

• A crest meeting a trough → they cancel out → dark zone

The result is an alternating pattern of bright and dark bands on the screen: that is an interference pattern.

What If We Did the Same With Matter?

In the 20th century, researchers then had an idea: apply Young’s experiment no longer by projecting photons (light) but electrons (matter). The experiment is therefore similar, but instead of a laser, an electron gun is used to then measure on the screen where the matter lands.

Obviously, with this experiment, we are going to get two bands of matter, right?

Well, still no! An interference pattern is observed as well. This result was not a complete surprise to everyone. In 1924, physicist Louis de Broglie had already theoretically proposed that matter, like light, could have a wave-like nature. But this time, it's not a wave-like light; it's a probability wave.

This is one of the greatest discoveries in quantum physics: at the atomic level, the particle has no defined position. The position of a particle is determined by a function called the wave function, ψ(x), which describes the probabilities of finding that particle at a given location in time.

A small clarification on this concept of undefined position to make sure the concept is clear, because this is the moment where our rational brain can start to “let go.”

Let’s take a coin for a coin toss. We throw it in the air and hide the result. We are in a state of uncertainty, but this uncertainty is called epistemic. We do not know the result (heads or tails) because we have not looked yet, yet that result already exists.

For a particle in the quantum world, the uncertainty is called ontological. It is not that we lack information about the position of the particle; it is that this position simply does not exist yet. This is what is called quantum superposition: an unmeasured particle exists in multiple states simultaneously.

However, measurement changes everything. When we measure the position of a particle, we will find it in one of the possible positions described by the wave function. We then say that the wave function “collapses” because it restricts the possibilities into a single real state.

As an analogy, it is a bit like Minecraft. A default Minecraft map is 60 million x 60 million blocks. For the initial loading, the server does not generate the entire map. It only generates the world around the observer, i.e., the player. However, when the player moves, they force the server to generate the world's continuation.

Where this analogy reaches its limits is that the generation of the Minecraft world, even if it is random, is still deterministic because each world has its own seed. The quantum world, on the other hand, appears to be purely random, meaning without hidden information.

Let’s return to Young’s experiment. What would happen if, when a particle passes through a slit, we placed a detector there to observe which slit the particle goes through? We recall that a wave passes through both slits at once.

This is the moment where the brain completely lets go: observing the particle changes the result of the experiment. Indeed, observing that particle “forces” it to have a defined position, and it then behaves like a classical marble. The result, therefore, gives us two bands of matter.

To summarize what we have seen so far: an unobserved particle exists as a probability wave, in multiple positions simultaneously. As soon as we measure it, this wave collapses, and the particle ends up at a precise location.

But then, why do we never see this in everyday life? The answer is decoherence.

Decoherence

Quantum superposition is only possible as long as a particle remains isolated from its environment. As soon as it interacts with anything, another atom, a photon, an electric field, that interaction constitutes a measurement in the quantum sense. The wave function collapses, and the particle ends up in a precise state.

An isolated electron in a vacuum can remain in superposition. But a macroscopic object like a table is made up of billions upon billions of atoms that permanently interact with the surrounding air, light photons, and electromagnetic fields. These interactions occur billions of times per second. The superposition collapses instantaneously before we can even observe it.

That is why quantum physics is only observable at the atomic scale. And that is also why a single electron in a transistor behaves very differently from an object we can hold in our hand.

The Key is Information

OK, so the original Young’s experiment with light produces an interference pattern because light is a wave. The variation with electrons (or indeed subsequently other elements such as atoms) also produces an interference pattern, which proves that matter is a wave, but this time a probability wave. When we measure the result, we change the result of the experiment because we force the particle to “choose” its position.

But incidentally, how does this measurement work in the experiment? It works thanks to photons. Indeed, when the electron passes through one of the slits, we project a photon which will interact with the electron and be re-emitted in a direction that allows us to deduce which slit the electron went through.

Researchers wanted to know what would happen if they performed the exact same experiment, measuring which slit the particle went through, but this time, instead of reading the information encoded in the orientation of the photon, they destroyed that information.

And here, another surprise: if we destroy the information, we return to an interference pattern. It was as if, since we were not using that information, there was nothing forcing the particle to choose which slit to go through, and so it could remain in the form of a probability wave.

This new experiment, therefore, demonstrates something fundamental in quantum physics: technically, it is not the act of measuring that influences the experiment, but whether or not this information exists somewhere in the universe. If this information is destroyed, the interference pattern returns.

The key is therefore information.

NOTE: How does the destruction of this information work? One might think it would simply be a matter of having the photon absorbed by an absorbing surface before reading it, but this does not work, and we are left with bands. Indeed, by doing so, the information theoretically exists because the absorbing surface could have determined the position of the particle through the orientation of the photon. The destruction works with another incredible principle of quantum physics that I will not detail in this article: entanglement. The photon is sent onto a special crystal, which splits it into two twin photons quantumly linked. One of the twins is then destroyed, making the information unrecoverable because to read the information, one absolutely needs to read both twins. To simplify, the two twins are not copies; they form a single system whose properties are not individually defined.

The Tunnel Effect

We are slowly getting closer to SSDs. But before that, there is one last quantum concept we need to talk about: the tunnel effect.

We said that an unobserved electron does not exist like a marble at a precise location. It exists as a probability wave spread out in space. This wave function gives a probability of finding the electron at each point in space.

Now let’s imagine a physical barrier. We send an electron toward this barrier. Classically, if the electron does not have enough energy to pass over it, it is blocked. Full stop.

Yet quantum mechanically, the wave function of the electron does not stop abruptly at the barrier. Because it is a wave, it propagates and gradually decays through the barrier. It does not fall to zero. On the other side, there therefore remains a non-zero probability of finding the electron.

This is the tunnel effect: a real chance for the electron to end up on the other side, without having had the classical energy needed to cross.

This probability is not fixed. It depends directly on the thickness of the barrier: the thinner the barrier, the more the wave function survives on the other side, and the higher the tunneling probability. At our scale, the barriers are far too thick for this effect to be observable. But at the scale of a few nanometers, the probability exists.

How SSDs Use Quantum Physics

In an SSD, we want to store data. They work with bits, but it is precisely in the management of these bits that the principles of quantum physics come into play.

In an SSD, each bit is encoded in cells called floating gates: small zones isolated on all sides by an insulating layer. This box can contain electrons or not:

• Box with electrons: Bit = 0

• Box without electrons: Bit = 1

If we need:

• To write, we therefore need to make electrons enter this isolated box. What we do is apply an electric voltage that deforms the wave function of the electrons and increases their probability of ending up on the other side. The electrons, therefore, cross the barrier via the tunnel effect.

• To erase, we apply a reverse voltage, which also impacts the wave function, and the electrons cross in the other direction.

• To read, it is a classical, non-quantum measurement: we measure the electric current passing through the transistor.

  • Electrons present: weak current: 0.

  • No electrons: strong current: 1.

We saw, however, that the wave function gives a probability, not a certainty. If we apply a voltage to write or erase, we therefore only have a probability that the electron will cross the barrier.

How can an SSD be reliable then? An individual electron is unpredictable, but we never send just one electron. We send millions simultaneously. Statistically, enough of them cross the barrier to charge the floating gate reliably.

And after each write, the controller immediately re-reads the cell to verify. If not enough electrons have crossed, it tries again. That is why SSDs embed error correction mechanisms, ECC (Error Correcting Code), precisely because the process is probabilistic by nature.

When a cell exceeds a certain error threshold over time, it is finally marked as defective and taken out of service. The data it held is moved to a healthy cell.

That is why SSDs always have an over-provisioning capacity: a reserve of cells invisible to the user, planned from the manufacturing stage to replace defective cells over time. And that is also why an SSD does not fail all at once; it degrades progressively, cell by cell, until the reserve is exhausted.

And this is where quantum physics imposes its limits. The more transistors shrink, the thinner the insulating barriers become, and the more the tunnel effect becomes uncontrollable, electrons escape spontaneously, errors increase, and cells age faster. Moore’s Law, which predicts a doubling of transistor density every two years, is today running up against these fundamental physical limits. This is not an engineering problem: it is quantum physics that sets the boundary.

Summary

• Matter is made up of atoms, themselves composed of a nucleus (protons and neutrons) and electrons. An atom is almost entirely empty: what we perceive as “solid” is an illusion created by the electromagnetic forces between atoms.

• Light is both an electromagnetic wave and a particle called a photon. In flight, it behaves like a wave, but it is emitted and absorbed like a particle, in one single hit, in one single place.

• Young’s double-slit experiment proves that light is a wave: it produces an interference pattern, impossible to obtain with classical particles.

• Matter behaves in the same way. But unlike light, its wave is not physical: it is a probability wave that describes the possible positions of a particle. This is quantum superposition: an unmeasured particle exists in multiple states simultaneously.

• It is not the act of measuring that collapses the superposition: it is the existence of the information somewhere in the universe. If the information is destroyed, the superposition is restored.

• Decoherence explains why we never see superposition at our scale: any macroscopic object permanently interacts with its environment, which instantaneously collapses its wave function.

• The tunnel effect is a direct consequence of the wave-like nature of particles: the wave function of an electron does not stop abruptly at a physical barrier. There exists a non-zero probability of finding it on the other side, without having had the classical energy to cross.

• SSDs exploit the tunnel effect to write and erase data: an electric voltage deforms the wave function of electrons and increases their probability of crossing the insulating barrier of a floating gate. Reliability rests on the large number of electrons sent and on ECC.

Resources

More From the Systems Category

• Instruction Pipelining

• Simultaneous Multithreading

• Linux Soft vs. Hard Lockup

Sources

• Something Deeply Hidden

• We Have No Idea

Explore Further

• Quantum Country

• The Double-Slit Experiment - Veritasium

❤️ If you enjoyed this post, please hit the like button.

💬 Did you know quantum physics was hiding in your laptop all along? I’d love to hear your reaction in the comments.

Leave a comment

Introduction

If you’ve been a subscriber since mid-2025, you were already here when I published the post that performed best on my newsletter: Working on Complex Systems.

I really loved writing it, and I always had in mind to revisit it at some point. So, when someone at ScyllaDB reached out to invite me to speak at Monster SCALE Summit, I saw the perfect opportunity to turn it into a talk.

The video isn’t a 1:1 mapping of the original content. I expanded it with more examples and new ideas. In it, I define what complex systems are, then discuss their common characteristics, and finally explore patterns for navigating them.

Hope you will enjoy it!

Video

Resources

More From the Distributed Systems Category

• Latency and User Experience

• Probabilistic Increment

• Bloom Filters

Explore Further

• Slides

• Monster SCALE Summit - 2026 Tech Talks

❤️ If you enjoyed this post, please hit the like button.

Leave a comment

At Google, I recently switched to a new domain: Google Distributed Cloud Connected1. Here, all the teams are very busy, and finding an efficient way to communicate over email or chat can be challenging, especially when asking someone to do something.

Recently, I came across a simple technique: three bullets and one call to action. The idea is the following:

• Add three bullet points explaining the key context

• Follow with one clear call to action

Let’s look at a concrete example. Suppose you receive the following email:

Hi,

I recently wrote a design doc on how to save storage in the context of X, where I describe the current problem and the approach we could take to address it. In the document, I go through the main trade-offs involved and explain why the proposal focuses on solution Y in particular. I also included several open questions related to the deployment strategy and some areas where feedback would be especially helpful. It would be great if you could take a look at the document and leave comments by Friday.

Thanks!

Quite a mouthful. It requires a non-trivial amount of brain time to understand both the context and what the person is actually asking for.

Now let’s apply the three bullets and a call to action strategy:

Hi,

• I recently wrote a design doc on how to save storage in the context of X.

• It highlights the main trade-offs and focuses on the solution Y.

• I’ve added open questions around the deployment strategy.

Could you please have a look and leave comments by Friday?

Thanks!

Much better, right? The call to action is clear, and the context is structured around short and easy-to-scan sentences.

Why does it work? When communicating via email or chat, people prefer short and memorable messages that do not require too much cognitive effort to process. Bullet points help break information into smaller chunks, which makes the message easier to scan quickly. Ideally, the bullet points and the call to action should be as short as possible.

Another aspect is that 3 is often a magic number in communication. With 2 items, you often get a contrast. With 3 items, you start to get a small structure or rhythm that is easier for the mind to process. That is one of the reasons why the rule of three appears so often in writing, storytelling, and presentations, where it helps make ideas more engaging and convincing.

Remember: to improve your chances of getting an answer to your request, use 3 short bullets and an efficient call to action.

Resources

More From the Beyond the Code Category

• 10 Rules I Learned About Technical Writing

• The XY Problem

• Don’t Forget About Your Mental Health

Sources

• Rule of three — Thinking Insights

• Rule of three (writing) — Wikipedia

Only 2, sorry about that.

❤️ If you enjoyed this post, please hit the like button.

💬 What do you think about this strategy? Have you tried something similar?

Leave a comment

Network Effects

Let’s consider a dating app. If there are only three people who installed the app in New-York, anyone new will probably try it for a few seconds and uninstall it. But if a big part of the city is on it, then someone single will probably stick around.

Said differently, the more people use the product, the more valuable it becomes. There’s a term for that type of product, and it’s called the network effect: when a product gets more valuable as more people use it.

Delving into the network effect, it’s not a single force but actually a trio of forces:

• The acquisition effect: How a product can use its own network to attract new people.

• The engagement effect: The more people join, the more useful and sticky it becomes.

• The economic effect: A larger network reduces costs, improves monetization, and strengthens the business model.

Now comes the real challenge. If our product relies on network effects, how do we launch it? Do we start from zero, or wait until enough people are on board? It’s a chicken-and-egg problem, and that’s the cold start problem.

Anti-Pattern: Big Bang

One common mistake to solve the cold start problem is the big bang launch: releasing to everyone before any community exists.

Google+ is the perfect example. Launched in 2011, it was Google’s attempt at a social network with a Facebook-style feed. The problem was that when people joined, they found empty timelines and left. Google later admitted that 90% of user sessions lasted less than five seconds.

At one point, Google even tied YouTube comments to Google+, requiring an account just to comment. The platform eventually reached more than 500 million users, but the issue was never sign-ups. The real problem was that a newcomer’s first session didn’t feel like walking into a lively room. It was forced growth instead of real networks.

Google+ didn’t fail because Google couldn’t build a social network. It lost because it never created a place where a new user could land in a live network. In short, Google+ failed to solve the cold start problem.

Pattern: Atomic Networks

A solution to the cold start problem was applied by Tinder, and it involves focusing on the concept of atomic networks.

Back then, dating apps were not very popular. Yet, Tinder was ambitious and wanted to succeed in that market. Instead of launching worldwide, they did the complete opposite. They organized a party at a college that required installing the application to attend. The next day, most students there had Tinder installed. This college was an atomic network: the smallest self-sustaining cluster of users where network effects actually work.

Soon after, they repeated the same process in another college in the same city, with the same result: within days, most students there had joined Tinder.

What’s powerful about atomic networks is simple: if we can build one network, we can build two. If we can build two, we can build thousands. Tinder repeated this strategy college by college, then city by city, eventually growing into entire countries.

The takeaway is that when we start a product with network effects, the first step is to build a single, tiny network that’s self-sustaining on its own. We just want to get started. If we can create one stable, engaged network, then we can build a second one next to it. From there, we can replicate the process and eventually connect them into one large network that spans the whole market.

Summary

• Network effects happen when a product gets more valuable as more people use it.

• How do we launch such a product with zero users? That’s the cold start problem.

• Big bang launches fail when newcomers find empty networks.

• One effective approach is to build atomic networks: the smallest self-sustaining clusters where network effects work.

• If we can build one atomic network, we can repeat it and scale across a market.

Resources

More From the Beyond Code Category

• Don’t Forget About Your Mental Health

• The XY Problem

• Lateral Thinking

Sources

• The Cold Start Problem

• Project Strobe: Protecting your data, improving our third-party APIs, and sunsetting consumer Google+

• Google+: Communities and photos

❤️ If you enjoyed the post, please consider giving it a like.

💬 The book is really great and covers many more aspects. I definitely recommend it. Are you into product management? What resources would you recommend?

Leave a comment

Join the Discord

☕ Welcome to The Coder Cafe! Today, we discuss a recent comment from Linus Torvalds about the use of a helper function. Get cozy, grab a coffee, and let’s begin!

In August 2025, there was (yet another) drama involving Linus Torvalds replying on a pull request:

No. This is garbage and it came in too late. I asked for early pull requests because I’m traveling, and if you can’t follow that rule, at least make the pull requests good.

This adds various garbage that isn’t RISC-V specific to generic header files.

And by “garbage” I really mean it. This is stuff that nobody should ever send me, never mind late in a merge window.

Like this crazy and pointless make_u32_from_two_u16() “helper”.

That thing makes the world actively a worse place to live. It’s useless garbage that makes any user incomprehensible, and actively WORSE than not using that stupid “helper”.

If you write the code out as “(a << 16) + b”, you know what it does and which is the high word. Maybe you need to add a cast to make sure that ‘b’ doesn’t have high bits that pollutes the end result, so maybe it’s not going to be exactly pretty, but it’s not going to be wrong and incomprehensible either.

In contrast, if you write make_u32_from_two_u16(a,b) you have not a f^%$ing clue what the word order is. IOW, you just made things WORSE, and you added that “helper” to a generic non-RISC-V file where people are apparently supposed to use it to make other code worse too.

So no. Things like this need to get bent. It does not go into generic header files, and it damn well does not happen late in the merge window.

Let’s not discuss the rudeness of this comment (it’s atrocious). Instead, let’s focus on the content itself.

, a popular newsletter, wrote a post about it:

the main point Linus makes here is that good code optimizes for reducing cognitive load. {…] Humans have limited working memory capacity - let’s say the human brain can only store 4-7 “chunks” at at time. Each abstraction or helper function costs a chunk slot. Each abstractions costs more tokens.

I share the view that good code optimizes for reducing cognitive load1, but I don’t understand Linus’s comment in exactly the same way.

Yes, Linus is virulent about the helper function, but in my opinion, his main argument isn’t simply that an abstraction costs a “chunk slot” as mentioned; it’s rather that this isn’t the right abstraction.

Here is the code added in the pull request:

/**
 * make_u32_from_two_u16 - return u32 number by combining
 * two u16 numbers.
 * @hi: upper 16 bit number
 * @lo: lower 16 bit number
 */
#define make_u32_from_two_u16(hi, lo)	(((u32)(hi) << 16) | (u32)(lo))

This macro builds a 32-bit integer by putting one 16-bit value in the high half and the other in the low half. For example:

hi     = 1010101111001101
lo     = 0001001000110100
result = 10101011110011010001001000110100
         |--------------||--------------|
               hi               lo

The main problem with this macro isn’t necessarily that it exists. It’s that its intent (meaning what it tries to accomplish) could have been clearer.

Indeed, the helper’s name doesn’t tell which word is high and which one is low and that’s exactly what Linus is calling out with “you have not a f^%$ing clue what the word order is”.

Because we can’t get the intent from the name (make_u32_from_two_u16), we have to open the macro to understand the order. That’s precisely why it costs a “chunk slot.”: not because the abstraction exists, but because it’s an ambiguous one.

If we wanted to keep using a macro, a better approach, in my opinion2, would be to encode the word order in the name itself (msw = most significant word, lsw = least significant word):

/**
 * make_u32_from_msw_lsw - return u32 number by combining
 * two u16 numbers in msw:lsw word order.
 * @msw: upper 16 bit number
 * @lsw lower 16 bit number
 */
#define make_u32_from_msw_lsw(msw, lsw) (((u32)(msw) << 16) | (u32)(lsw))

In this case, the word order is carried by the macro name, which makes it a clearer abstraction. Reading the call site doesn’t require opening the macro to understand the word order:

u32 v = make_u32_from_msw_lsw(0xABCD, 0x1234);

Such an abstraction doesn’t cost a “chunk slot” in terms of cognitive load. Its intent is clear from the name, so we don’t need to load an extra piece of information into our working memory to understand it.

In summary, if we want to optimize for cognitive load, there’s not necessarily an issue with using helper functions. But if we do, we should make the abstraction as explicit as possible, and that starts with a clear function name that conveys what it tries to accomplish.

Resources

More From the Programming Category

• Readability

• Cognitive Load

• Nested Code

Sources

• Re: [GIT PULL] RISC-V Patches for the 6.17 Merge Window, Part 1 - Linus Torvalds // The discussion.

• GitHub // The code proposed in the pull request

Explore Further

• Linus and the two youts // Interestingly, the macro was plain wrong when the second word was negative. The full explanation is here.

❤️ If you enjoyed this post, please hit the like button.

💬 Where do you draw the line between “helpful” and “harmful” abstraction?

Leave a comment

We’re in 2022. It’s Saturday evening, and I’m about to go to bed. I’m on-call that night. I haven’t been paged, but just to make sure everything is OK, I logged in and checked Slack.

An incident was going on, and a colleague was already on it. I DMed him: “Why didn’t you contact me?”

He replied: “It’s late and I thought you might be sleeping. I was awake, so I looked to see if there’s something I could do.”

My first reaction was: I’m on-call. I’m paid for it. I’ll take care of it. Go to bed.

But here’s the thing: on a Saturday evening, he chose to help because he thought I might be sleeping, even though I was the one on-call, the one paid to handle it.

That was a pure act of kindness. No points. No credit. Just care. And after that? Honestly, I would have done anything for that person.

Why Kindness Wins

At work, we work with people long before we work with code. There’s always a little distance between us: roles and power dynamics, deadlines and pressure, different cultures, communication styles, sometimes different time zones. Kindness is the fastest bridge across that distance.

Kindness is about being generous, considerate, and having concern for others without expecting praise or reward in return. It’s a voluntary act that creates psychological safety among team members. When people feel safe, they surface risks earlier, ask the “naive” questions, and move faster together.

Kind people make work better day in, day out. Kindness boosts trust, speeds decisions, reduces stress, and quietly raises the bar for everyone.

Let’s look at a few places where kindness matters in our daily jobs:

• Code review: When we’re assigned a review, we’re not there to rate someone’s code. We’re there to merge the best possible change together. Be respectful and stay factual. Favor questions over pronouncements: “What scenarios does this handle? I’m worried about X; would Y cover Z?” Point out what’s good, suggest concrete fixes, and link to standards or examples. If there’s confusion, offer help.

• Meetings: Make space so everyone can be heard. Don’t interrupt. Invite quieter people in: “Ben, anything you would add?” It’s not because someone is more vocal that they’re more right.

• Mentoring: People make mistakes. Don’t jump to blame or perform expertise. The goal is to protect in public and correct in private. Give clear, kind feedback, focus on the next step, and share your own past mistakes to lower the temperature.

• Random thank you: When you receive help or just enjoy working with someone, say thank you. Recognition matters, and doing it publicly multiplies the effect. For example, at Google, there’s a program called gThanks that lets you thank someone publicly so others can see it too.

• Make time to listen: Being kind also means making time to listen. I remember going through a difficult period of my life, and a former manager just took time to talk, without judging. That mattered more than any advice.

• Self-compassion: Kindness also applies to yourself. Give yourself the same understanding you would give a teammate. Take breaks, ask for help, forgive your own mistakes, and learn from them.

Being kind is a bridge to people, and even in a professional context, as Aesop wrote, no act of kindness, no matter how small, is ever wasted.

Resources

More From the Beyond the Code Category

• Don’t Forget About Your Mental Health

• Keeping a Mistake Journal

• The XY Problem

Sources

• Why Kindness at Work Pays Off

Explore Further

• Random Acts of Kindness Foundation

❤️ If you enjoyed this post, please hit the like button.

💬 What’s one act of kindness that changed your workday?

Leave a comment

🎃 Welcome to The Coder Cafe! Today, we examine the Therac-25 accidents, where design and software failures resulted in multiple radiation overdoses and deaths. Make sure to check the Explore Further section to see if you’re able to reproduce the deadly issue. Get cozy, grab a pumpkin spice latte, and let’s begin!

Therac-25

Treating cancers used to require a mix of machines, depending on tumor depth: shallow or deep. In the early 1980s, a new generation promised both from a single system. That was a big deal for hospitals: one machine instead of several meant lower maintenance and fewer systems to manage.

That was the case with the Therac-25.

The Therac-25 offered two therapies with selectable modes:

• Electron beam: Low-energy electrons for shallow tumors (e.g., skin cancer).

• X-ray photons: High-energy radiation for deep tumors (e.g., lung cancer).

Earlier Therac models allowed switching modes with hardware circuits and physical interlocks. The new version was smaller, cheaper, and computer-controlled. Less hardware and fewer parts meant lower costs.

However, what no one realized soon enough: it also removed an independent safety net.

The Horror Begins

On a routine day, a radiology technologist sat at the console and began entering a plan:

• By habit, she selected X-ray (deep mode).

• Then she immediately corrected it for Electron (shallow mode) and hit start.

The machine halted with a MALFUNCTION 54 message. The operator’s manual didn’t explain the code. Service materials listed the number but gave no useful guidance, so she resumed and triggered the radiation.

The patient was receiving his ninth treatment. Immediately, he knew something was different. He reported a buzzing sound, later recognized as the accelerator pouring out radiation at maximum. The pain came fast; paralysis followed. He later died from radiation injury.

Weeks later, a second patient endured the same incident on the same model.

What Happened?

Initially, the radiology technologist entered ‘X’ for X-ray (‘▮’ is the cursor and ‘…’ are other fields):

MODE:              X
...                ▮
BEAM READY:        

She immediately hit Cursor Up to go back and correct the field to ‘E’:

MODE:              E
...                ▮
BEAM READY:        

After a rapid sequence of Return presses, she moved back down to the command area:

MODE:              E
...                
BEAM READY:        ▮

From her perspective, the screen showed the corrected mode, so she hit return and started the treatment:

MODE:              E
...                
BEAM READY:        ▮<Enter>

Behind the scenes, the Therac-25 software ran several concurrent tasks:

• Data-entry task: Monitored operator inputs and edited a shared treatment-setup structure.

• Hardware-control task: On a periodic loop, snapshotted that same structure and positioned the turntable and magnets based on user input.

Because both tasks read the same memory with no mutual exclusion, there was a short window (on the order of seconds) in which the hardware-control task used a different value than the one displayed on the screen.

As a result:

• The UI showed Electron mode, which looked correct to the operator.

• The hardware-control task had snapshotted stale data and marked the system as ready even though critical elements (e.g., turntable position, scanning magnets/accessories) were not yet aligned with electron mode.

• When treatment was started, the machine delivered an effectively unscanned, high-intensity electron beam, causing a massive overdose.

This is a race condition example: the outcome depends on the timing of events, here, the input cadence of the technologist. Depending on the timing, the system could enter a fatal state, with one process seeing ‘X’ while another saw ‘E’.

The manufacturer later confirmed the error could not be reproduced reliably in testing. The timing had to line up just right, which made the bug elusive. They initially misdiagnosed it as a hardware fault and applied only minor fixes. Unfortunately, the speed of operator editing was the key trigger that exposed this software race.

Another Tragedy

The problem could have stopped here, but it didn’t.

Months later, another fatal overdose occurred, this time caused by a different software defect. It wasn’t a timing race. This time, the issue was a counter overflow within the control program.

The software used an internal counter to track how many times certain setup operations ran. After the counter exceeded its maximum value, it wrapped back to zero. That arithmetic overflow created a window where a critical safety check was bypassed, allowing the beam to turn on without the proper accessories in place.

Again, the Therac-25 fired a high-intensity beam without the proper hardware configuration.

Conclusion

Both the race condition and the counter overflow stemmed from the same design flaw: the belief that software alone could enforce safety. The Therac-25 showed, in tragic terms, that without independent safeguards, small coding errors can have catastrophic consequences.

We should know that whether it’s software, hardware, or a human process, every single safeguard has inherent flaws. Therefore, in complex systems, safety should be layered, as illustrated by the Swiss cheese model:

In total, there were six known radiation overdoses involving the Therac-25, and at least three were fatal.

Resources

More From the Reliability Category

• Adaptive LIFO

• Resilient, Fault-tolerant, Robust, or Reliable?

• Lurking Variables

Sources

• The Worst Computer Bugs in History: Race conditions in Therac-25

• Killed By A Machine: The Therac-25

• An Investigation of the Therac-25 Accidents

Explore Further

I created a Docker image based on a C implementation from an MIT course simulating the operator console of the Therac-25 interface:

You can run the UI using Docker:

 docker run --rm -it -e TERM=xterm-256color teivah/therac-25

Simulator commands:

• Beam Type: ‘X’ or ‘E’

• Command: ‘B’ for beam on, or ‘Q’ to quit the simulator.

👉 Try to trigger the MALFUNCTION 54 error based on the scenario discussed.

❤️ If you enjoyed this post, please hit the like button.

💬 Any other horror coding stories you want to share?

Leave a comment

We often celebrate teams for moving fast. But speed alone can be a trap. A rush of fast changes that barely move the product toward the real goal shouldn’t count as a win. When we talk about team productivity, we should understand that speed ≠ velocity:

• Speed is how quickly a team ship changes.

• Velocity is speed with direction, the movement toward a defined goal.

Let’s look at three teams to illustrate these definitions.

• Team A has ideal velocity. Each iteration is represented by an arrow: its length shows speed, and its angle shows direction. Every iteration moves the team consistently closer to the goal.

  

• Team B has a speed problem. Each iteration is well-aligned (correct angle), but the team delivers too slowly (small length), resulting in a lower velocity. They finish only halfway to the goal.

  

• Team C ships as rapidly as team A (same arrow length). However, various factors such as frequent bug fixes and changing targets make their direction inconsistent. Despite high speed, their velocity is low, and they end only halfway to the goal, just like team B.

  

As we can see, velocity requires both speed and direction. A team moving too slowly or in inconsistent directions will make little progress, even if they’re busy. Only when speed is high and direction is aligned do teams reach their goals efficiently.

Measuring team speed isn’t useless, though. We can track speed by considering various metrics such as deployment frequency, average time in code review, or mean time to recovery (MTTR) following a production bug. These metrics are interesting to track and provide a certain perspective to understand a team's productivity. Yet, speed shouldn’t be the sole dimension to track.

The danger of tracking speed only is that a team might become organized in a way to optimize short-term delivery. The team might focus on delivering many changes that, together, do not move the product in a meaningful way. Instead, teams should track speed and velocity.

As we said, velocity is speed with direction. We already discussed metrics to track speed; what is missing is monitoring direction.

Setting clear and factual objectives that align with the business strategy helps us track direction. For example:

• Payment success rate above 99 percent.

• Signup to activation above 50 percent within 7 days.

• Retention after week 4 is above 40 percent.

The easier a metric is to measure, the easier it is to track the direction over time.

One caveat is how to report progress. In a past team, we used to set OKRs (Objectives and Key Results) every semester. Some objectives were difficult to measure, so we tracked progress differently. Say we created 50 tickets and closed 45. In that case, we reported that we reached 90% of the OKR. That number said nothing about real progress toward the objective. Key results should be outcomes, not ticket counts.

Something else to mention, I read here and there that we should track velocity by counting the number of story points delivered within a timeframe (e.g., during a sprint). I strongly disagree with this. Let me give an example:

• A team ships a first change ← 3 story points

• The team finds a bug and ships a fix ← 2 story points

• Later, the team learns the initial approach is not the best, so it ships another change ← 5 story points

On paper, the team delivered 10 story points. That is only speed, though, not velocity. If we care about direction as well, the team only delivered a single feature. Story points measure effort; they don’t measure progress toward the objective.

Summary

• Speed is how fast we ship. Velocity is speed with direction toward a goal.

• The goal of a team shouldn’t be to reach high speed; it should be to reach high velocity, where rapid iterations translate into real system-level improvements.

• Tracking story points, for example, is a measure of speed, not velocity.

• Objectives tracking should use outcomes, not ticket counts. Reporting 90 percent of tickets done is not a good measure of progress toward the objective.

Resources

More From the Beyond Code Category

• Keeping a Mistake Journal

• Streetlight Effect

• Survivor Bias

Sources

• Sprint Velocity in Scrum: How to Measure and Improve Performance // The velocity definition I disagree with.

❤️ If you enjoyed this post, please hit the like button.

💬 What’s your take on speed vs. velocity? How do you measure velocity in your team?

Leave a comment

CRDTs, short for Conflict-Free Replicated Data Types, are a family of data structures built for distributed systems. At first sight, CRDTs may look intimidating. Yet at their core, the idea is not that complex. What makes them special is that they allow updates to happen independently on different nodes while still guaranteeing that all replicas eventually converge to the same state.

To understand how CRDTs achieve this, we first need to step back. We need to talk about concurrent operations and what coordination means in a distributed system. Let’s take it step by step.

Concurrent Operations

What does concurrent operations mean? Our first intuition might be to say they happen at the same time. That’s not quite right. Here’s a counterargument based on a collaborative editing example.

1. While on a plane, Alice connects to a document and makes an offline change to a sentence.

2. An hour later, Bob connects to the same document and edits the very same sentence, but online.

3. Later, when Alice lands, both versions have to sync.

The two edits (1. and 2.) were separated by an hour. They didn’t happen at the same time, yet they are concurrent.

So what’s a better definition for concurrent operations? Two operations that are not causally related.

In the previous example, neither operation was made with knowledge of the other. They are not causally related, which makes them concurrent. Yet, if Bob had first seen Alice’s update and then made his own, his edit would depend on hers. In that case, the two operations wouldn’t be concurrent anymore.

We should also understand concurrent ≠ conflict:

• If Alice fixes a missing letter in a word while Bob removes the whole word, that’s a conflict.

• If Alice edits one sentence while Bob edits another, that’s not a conflict.

Concurrency is about independence in knowledge. Conflict is about whether the effects of operations collide.

Coordination

Now, let’s talk about coordination in distributed systems.

Imagine a database with two nodes, node 1 and node 2. A bunch of clients connect to it. Sometimes requests go to node 1, sometimes to node 2. Let’s say two clients send concurrent and conflicting operations:

In this case, we can’t have node 1 storing $200 while node 2 stores -$100. That would be a consistency violation with the two nodes disagreeing on Alice’s balance.

Instead, both nodes need to agree on a shared value. To do that, they have to communicate and decide on one of the following:

• Reject both operations

• Accept client A’s update and set the balance to $200

• Accept client B’s update and set the balance to -$100

The very action of nodes communicating and, if needed, waiting to agree on a single outcome is called coordination.

Coordination is one way to keep replicas consistent under concurrent operations. But coordination is not the only way. That’s where CRDTs come in.

CRDT

CRDT stands for Conflict-Free Replicated Data Types. In short, CRDTs are data structures built so that nodes can accept local updates independently and concurrently, without the need for coordination.

If you read our recent post on availability models, you might notice we’re now in the territory of total availability: a system is totally available if every non-faulty node can execute any operation. Total availability comes with weaker consistency. For CRDTs, the consistency guarantee is called Strong Eventual Consistency (SEC).

For that, CRDTs rely on a deterministic conflict resolution algorithm. Because every node applies the same rules, all replicas are guaranteed to eventually converge to the same state.

G-Counter

Let’s make this more concrete with a classic CRDT: the G-Counter (Grow-Only Counter).

Imagine a database with two nodes tracking the number of likes on a post. Node 1 receives a new like, increments its counter, and replies success to the client:

Then, node 1 communicates with node 2 to send this update:

Ultimately, both nodes converge to the same value: 6.

How does the conflict resolution work for a G-Counter?

Each replica keeps a vector of counters, with one slot per node. In our example, the total number of likes is 5. Let’s say node 1 has seen 2 likes and node 2 has seen 3 likes. So the initial state is the following:

Node 1: [2, 3]
Node 2: [2, 3]

When node 1 receives a new like, it only increments its own slot. Node 2 is now temporarily out of sync:

Node 1: [3, 3]
Node 2: [2, 3]

During synchronization, both nodes merge their vectors by taking the element-wise maximum:

[max(3,2), max(3,3)] = [3, 3]

Now both replicas converge to the same state:

Node 1: [3, 3]
Node 2: [3, 3]

The beauty of this algorithm is that it’s deterministic and order-independent. No matter when or how often the nodes sync, they always end up with the same state.

NOTE: Do you know Gossip Glomers? It’s a series of distributed systems challenges we briefly introduced in an earlier post. Challenge 4 is to build a Grow-Only Counter. It’s worth checking out if you haven’t already.

PN-Counter

CRDTs can also be combined to make a more complex CRDT. For example, if we want to track both likes and dislikes, we can use two G-Counters together. This data type is called a PN-Counter (Positive-Negative Counter).

Imagine two clients act concurrently on the same post: one likes it, another dislikes it.

The nodes exchange their updates and converge to the same value:

In the case of a PN-Counter, the conflict resolution algorithm is similar to the G-Counter. The difference lies in the fact that it involves not one but two vectors: one for increases and one for decreases.

Assume an initial state where node 1 has received 2 likes and 0 dislikes, and node 2 has received 3 likes and 0 dislikes:

-- Increase
Node 1: [2, 3]
Node 2: [2, 3]

-- Decrease
Node 1: [0, 0]
Node 2: [0, 0]

Now, suppose node 1 receives a new like and node 2 receives a dislike. Before the sync, the state is the following:

-- Increase
Node 1: [3, 3]
Node 2: [2, 3]

-- Decrease
Node 1: [0, 0]
Node 2: [0, 1]

When the replicas exchange their state, the merge rule is element-wise maximum for each vector:

-- Increase
[max(3,2), max(3,3)] = [3, 3]

-- Decrease
[max(0,0), max(0,1)] = [0, 1]

After sync, both nodes converge to:

-- Increase
Node 1: [3, 3]
Node 2: [3, 3]

-- Decrease
Node 1: [0, 1]
Node 2: [0, 1]

The final counter of likes is:

node1Likes + node2Likes - node1Dislikes - node2Dislikes
= 3 + 3 - 0 - 1
= 5

Use Cases

Let’s pause for a second. Based on what we’ve discussed, can you think of some use cases for CRDTs? A data structure where nodes are updated independently, concurrently, without coordination, and still guarantees that they converge to the same state?

One main use case is collaborative and offline-first systems. For example, Notion, a collaborative workspace, recently introduced a feature that lets people edit the same content offline. They rely on CRDTs, and more specifically on Peritext, a CRDT for rich-text collaboration co-authored by multiple people, including .

Another big use case is totally available systems that put availability ahead of strong consistency. As we’ve seen, nodes don’t need to coordinate before acknowledging a client request, which makes the system more highly available.

Take Redis, for example. It can be configured in an active-active architecture with geographically distributed datacenters. Clients connect to their closest cluster and get local latencies without waiting for coordination across distant regions. And yes, this setup is built on CRDTs.

We could also think about other applications for CRDTs, like:

• Edge & IoT: Devices update offline and merge later without a central server.

• Peer-to-peer: Peers share changes directly and match up when they reconnect.

• CDN/edge state: Keep preferences, drafts, or counters near users and sync to the origin later.

Different Types of CRDTs

There are two main types of CRDTs:

• State-based CRDTs: Convergence happens by propagating the full state.

• Operation-based CRDTs: Convergence happens by propagating the update operations.

State-Based CRDTs

In the previous examples, we looked at two state-based CRDTs: the G-Counter (Grow-Only Counter) and the PN-Counter (Positive-Negative Counter). In both cases, what was exchanged between the nodes was the entire state. For example, node 1 could tell node 2 that its total number of likes is 3.

With state-based CRDTs, states are merged with a function that must be:

• Commutative: We can merge in any order and get the same result.

• Idempotent: Merging something with itself doesn’t change it.

• Associative: We can merge in any grouping and get the same result.

Each synchronization monotonically increases the internal state. In other words, when two replicas sync, the state can only move forward, never backward. This is enforced by a simple “can’t-go-backwards” rule (a partial order), where merges use operations like max for numbers (as we’ve seen) or union for sets.

Operation-Based CRDTs

In operation-based CRDTs, nodes share the operations rather than the full state. Convergence relies on three properties:

• Commutativity of concurrent operations

• Causality: Either carried in the operations’ metadata (for example, vector clocks) or guaranteed by the transport layer through causal delivery

• Duplicate tolerance: Handled by idempotent operations, unique operation IDs with deduplication, or a transport layer that guarantees no duplicates

One example of an operation-based CRDT is the LWW-Register (Last-Writer-Wins Register), which stores a single value. Updates are resolved using a logical timestamp (such as Lamport clocks) along with a tie-breaker like the node ID. When a node writes a value, it broadcasts an operation (value, timestamp, nodeID). On receiving it, a node applies the update if the pair (timestamp, nodeID) is greater than the one it currently holds.

CRDT Types Summary

To summarize:

• State-based CRDTs:

  • Convergence is guaranteed because merging states is associative, commutative, and idempotent.

  • Don’t require assumptions on the delivery layer beyond eventual delivery.

  • Simpler to reason about.

  • Exchanging full states can be more bandwidth-intensive.

• Operation-based CRDTs:

  • More bandwidth-efficient; we only send the operations, not the whole state.

  • Correctness usually depends on having causal order (or encoding causality in the ops) and tolerating duplicates via idempotence/dedup.

  • More complex to implement (causal broadcast, vector clocks, or equivalent).

For completeness, there’s also a third type we should be aware of: delta-based CRDTs. Here, convergence is achieved by sending and merging fragments of state (deltas) rather than the entire state. A quick analogy to picture the differences:

• State-based CRDT: “From time to time, send me the whole document.”

• Operation-based CRDT: “When you make a change, tell me exactly what you did.” → “Adding word `miles` at position 42.”

• Delta-based CRDT: “When you make a change, send me just the delta that reflects it (for example, the updated sentence)” → “And miles to go before I sleep.”

CRDT vs. OT

We talked about collaborative document editing. So you might assume a system like Google Docs is based on CRDTs, right?

Well, that’s not the case. Google Docs is based on another concept called OT (Operational Transformation). The goal of OT and CRDT is the same: convergence among all nodes in a collaborative system.

The main difference is that OT requires all communication to go through the same server:

We haven’t mentioned it until now (on purpose), but with CRDTs, there’s no need for a central server to achieve convergence.

Back to our collaborative editing tool: if Alice and Bob are both offline but manage to connect their laptops directly, they could still achieve convergence without talking to a central server:

As we saw earlier, CRDTs embed a deterministic conflict resolution algorithm. The data type itself ensures convergence.

That’s the key difference: CRDTs don’t need to make any assumptions about the network topology or about a central server. considers CRDT to be the natural successor of OT.

NOTE: So, why is Google Docs still based on OT? Historical reasons. Google Docs was launched before CRDTs existed, and it still works really well. There’s no practical reason for Google to migrate from OT to CRDT, despite some discussions about it in the past.

Conclusion

• Operations are concurrent when they aren’t causally related; concurrency doesn’t automatically mean conflict.

• Coordination is when replicas communicate and, if needed, wait to agree on a single outcome for concurrent updates before acknowledging clients, so they don’t diverge.

• CRDTs accept independent updates on each replica and still converge via deterministic merge rules.

• Three types: state-based (share full state), operation-based (share operations), delta-based (share just the changed parts).

• CRDTs are a great fit for systems like offline-first collaboration and highly available systems.

• Unlike OT, CRDTs don’t rely on a central server to reach the same result everywhere.

Resources

More From the Distributed Systems Category

• Exploring Database Isolation Levels

• Consensus

• Safety and Liveness

Sources

• Ivan Zhao (Notion’s CEO) tweet on the new Notion offline collaboration feature

• Diving into Conflict-Free Replicated Data Types (CRDTs) - Redis

• CRDTs: The Hard Parts by

Explore Further

• Hacker News discussion

• Peritext - A CRDT for Rich-Text Collaboration

• Active-Active geo-distribution (CRDTS-based) - Redis

• Bartosz Sypytkowski’s 12-part blog series on CRDT

• crdt.tech

❤️ If you enjoyed this post, please hit the like button.

💬 Have you worked with CRDTs before, or do you see another use case where they shine? Share your thoughts in the comments!

Leave a comment

A few months ago, I was in a meeting discussing the state of a codebase and heard myself say:

After reflecting on my own words, I began questioning myself: I don’t even know what organic growth really means and whether it is ultimately positive or negative.

Discussing the concept with various colleagues, I noticed we had different interpretations. I also created a poll on X that went viral (#irony), showing that most people see organic growth as something positive.

If Perplexity is correct, the term organic growth was used first in Bushido: The Soul of Japan, published in 1899. The author, Inazo Nitobe, describes Bushido (aka the way of the samurai) as something that developed through organic growth rather than being the invention of a single person or the result of a single event. In other words, Bushido emerged gradually over decades and centuries as a collective experience, not the product of one individual.

It’s easy to draw parallels with the many codebases we maintain at work. The older the codebase and the more developers it has had, each with their own vision of how the code should look. Differences in perception arise from factors like developers’ varied mental models based on the parts of the code they know best.

Developers come and go, making changes, introducing biases about what the future should be, and after years, the codebase reflects hundreds or even thousands of such incremental changes.

This is organic growth in software: incremental delivery driven by agile methodology that shapes codebases, often in a bottom-up manner.

So, is organic growth good or bad? Let’s use the garden metaphor: imagine a public garden where anyone can plant whatever they want and wherever they want. Over time, sure, it’s colorful, but there’s little harmony, with a lack of overall guidance or direction:

Now imagine we control what can be planted and where, and periodically review and rearrange what’s growing. Our garden will likely look harmonious and well-maintained:

In my opinion, organic growth carries a somewhat negative connotation, implying a lack of direction, with systems growing out of ad hoc solutions and quick fixes rather than intentional design.

So, how can we move from a messy garden to a harmonious one? By transitioning from organic to controlled growth.

Controlled growth is the deliberate and reflective process of evolving a codebase. With controlled growth, we should also embrace progress that is not always perfect but consistently guided by shared patterns and standards, fostering sustainable, manageable, and harmonious development over time.

Here are five main rules for controlled growth:

• Be consistent. Follow agreed conventions and standards to build maintainable and predictable codebases. Many standards aren’t explicit, so strive to follow even the implicit ones. For example, if an entity is consistently named X, don’t be a cowboy and name it Y; stick to X.

• Plan ahead for large changes. Think strategically and collectively about big transitions such as significant new features. Avoid chaos and technical debt by breaking ambitious changes into smaller phases, communicating plans early, and leveraging documentation assets such as design docs (see Explore Further section).

• Hold regular retrospectives. After major development milestones, gather the team to review what went well and what could improve. Retrospectives foster a culture of learning and continuous improvement, a strategic pillar of controlled growth.

• Apply the Boy Scout rule. Frequent changes are inevitable and often rushed under pressure. Remember: “Always leave the campground cleaner than you found it.” When making a pull request, spend a little time refactoring small bits, cleaning legacy code, removing dead code, or renaming variables. This keeps the garden harmonious over time.

• Build a team where everyone shares responsibility and speaks openly. Controlled growth shouldn’t fall only on tech leads; it should be everyone’s job. When all team members feel heard and share responsibility for the health and quality of the codebase, the team grows on a stronger, more sustainable foundation.

Resources

More From the Programming Category

• Tidy First? // We already discussed organic growth in this post.

• Focus on Product Ideas, Not Requirements

• Cognitive Load

Sources

• Bushido: The Soul of Japan

Explore Further

• Design Docs at Google

❤️ If you enjoyed this post, please hit the like button.

💬 When you say a codebase grew organically, what do you mean?

Leave a comment

A Decade of Failures

For over a decade, I failed miserably at keeping an effective note-taking system, mostly for two reasons:

1. I never had a single, centralized place. I used to rely on a combination of Notion + Apple Notes + a physical notebook + Kindle highlights + Anki.

2. I didn’t have a system that was generic enough. I had a different format pretty much every time: book summaries, posts, computer science notes, mistake journal, personal growth notes, and so on.

The result was a messy system that didn’t scale. I kept losing knowledge, and it made learning inefficient. When we read or watch a great resource but don’t capture what we learned from it, chances are high that within a few months or even weeks, that knowledge fades away. That’s what kept happening to me.

In a world full of content—social media, work, books, courses, podcasts—being able to extract and retain working knowledge isn’t optional; it’s necessary if we want to keep growing as an engineer and as a person.

Let’s discuss an approach to solve this problem.

Second Brain

One of the first sentences in the book hooked me:

Our brain is great for processing ongoing tasks, but it’s not meant to retain everything. Over time, it lets go of unused ideas to make room for more relevant ones. This process, referred to as synaptic pruning, helps us adapt, but it also means we lose what we don’t externalize.

That’s the promise of a second brain: a place to offload notes and thoughts, avoid losing knowledge, and build on it over time with new ideas.

Before switching to the second brain approach, I also struggled with my notes because I wanted everything to be perfect. That drained a lot of time and energy. But one paragraph from the book really shifted my perspective:

That changed everything for me. I stopped chasing perfection and started building something that simply works, something that supports me every day, both at work and in my personal life.

What I found enriching is that the book presents not just a system, but also a mindset. Let’s start with the latter.

Mindset

Knowledge work is about taking information and turning it into results, for example, delivering on a project. All day, we consume and then produce:

The problem with this approach is that most of the information that we will gain will eventually be lost. Sure, we might remember the most important parts for a while. But what about the rest? It will fade, eventually.

What we miss is a feedback loop, a way to recycle information into knowledge that we can reuse later:

That’s what a second brain gives us: a way to turn information into assets we can reinvest in the future.

Another interesting idea in the book is to stop seeing notes as a flat list of things we’ve saved. Our notes can become building blocks, meaning pieces that help us create new ideas later on.

That’s a powerful shift: notes aren’t just storage, they’re raw material for thinking. They can connect and evolve into new ideas. Being able to track down all your notes in a single place makes those connections easier to spot and build on.

One last point I wanted to discuss: creativity. Here’s the definition given by Neuroscientist Nancy C. Andreasen:

A second brain is not only a memory tool, but also a thinking tool. Keeping an effective system to track down notes and ideas becomes a great tool to improve our creativity. Remember: creativity is not a talent, it’s a way of operating.

NOTE: Did you enjoy this punchline? It’s coming from a note I hold on creativity in my second brain 😎.

System

In this section, we’ll delve into what a note really is, how to take notes effectively, and then go over the PARA system presented in the book.

Notes

So, what’s a note? The author defines it this way:

Let’s go over the different parts of that definition.

First, a note is something to use, not just to collect. Again, we’re not building an encyclopaedia, we’re building something that works for us. For example, if we’re interested in public speaking, we don’t need a note on every single resource we read or watch.

Instead, the author suggests asking a few questions to decide what’s worth capturing:

1. Does it inspire us?

2. Is it useful?

3. Is this personal?

4. Is it surprising?

Ultimately, we should capture what resonates.

For example, we may have read this amazing book that everyone is talking about and discover that it doesn’t resonate with us at all1. Conversely, we may have watched a 30-second video that had a profound effect on us and triggered emotions. In this case, it’s better to spend some time creating a note on this video rather than the book.

When something resonates with us, it’s our emotion-based, intuitive mind telling us it’s interesting before our logical mind can explain why.

How to Take Notes

Every time we take a note, we should ask: “How can I make this as useful as possible for my future self?”

One way to do that is to be mindful of our future limited time. Instead of tracking dozens, if not hundreds, of lines, we should focus on finding the essence, meaning the heart and soul of what a resource is trying to communicate.

But in some cases, that’s barely possible. For example, I recently read the F1: A distributed SQL database that scales whitepaper. How can we capture the essence of such a dense technical document containing so much valuable information?

The solution, brought again by the author, is to use the progressive summarization technique. In short, it’s about layering our notes:

• Layer 1 - Captured notes: Either a copy and paste or even better writing, down in our own words what we understood from it.

• Layer 2 - Bolded passages: Go over the captured notes and mark in bold the most important pieces.

• Layer 3 - Highlighted passages: Go over the bolded passages and highlight the most important pieces.

• Layer 4 - Summary: Write down a summary.

Here’s an example from my F1 whitepaper notes. I created two main sections: Summary and Highlights.

• Layer 1: In Highlights, I captured all the raw content, for example, on locking.

  

• Layer 2: I bolded the passages that were most interesting to me.

  

• Layer 3: I highlighted the most important parts.

  

• Layer 4: I wrote a summary section with the ideas that were the most interesting for me.

  

In the end, the note looks like this:

# Summary

- ...
- ...
- ...

# Highlights

## Architecture

## Data model

## Locking

## Etc.

This approach allows me to come back later and start with a quick summary to refresh the main ideas. If I need a bit more, I can scan the bolded and highlighted passages. And if I really want to dive deeper, I can go through all the highlights.

Of course, every layer is optional. If we consider that a resource only needs layers 1 and 4 of summarization, or just 1 and 2, that’s perfectly fine.

Again, our second brain should be something that works for us.

PARA

The PARA system is at the core of the book. It’s a proposition on how to organize our notes. It is designed for actionability, with layers of action. To make things clear, a note can be assigned to one of the following domains:

• Project: A short-term effort with a possible due date and a clear outcome that needs to happen in order to mark the project as complete. For example, publish a blog post about second brain.

• Area: Ongoing responsibilities, what we are committed to, and what requires constant attention.

• Resource: A catchall for anything that doesn’t belong to a project or an area.

• Archive: When a note becomes inactive or outdated, we can move it to the archive.

NOTE: The distinction between area and resource wasn’t immediately clear to me, so here’s how I think about it.

I enjoy both fitness and climbing. But I’m only committed to fitness. I try to eat healthy, work out regularly, and so on. Climbing, on the other hand, is something I enjoy, but I only go from time to time. So in my system, fitness is an area, and climbing is a resource.

The PARA system has two main benefits.

• Clear focus: We’re not mixing short-term efforts with long-term maintenance. It helps us focus on outcomes and next steps rather than just piles of information.

• Genericity: PARA can handle all kinds of notes. It organizes information based on how actionable it is, not what kind of information it is.

As I said, I used to lack a system that was generic enough to track all my notes. Now, whether it’s a book I read, a course I followed, or a post I came across, I capture everything that resonates with me using the PARA system.

Notion

To implement the second brain, I used Notion, which I think is a fantastic tool with a lot of flexibility and configuration options.

If you don’t want to build your own second brain from scratch, you can check out this online tutorial:

Or, you can use my personal Notion template. This is the setup I use every day to track notes, growth tasks, my to-do list, areas, resources, and more.

Conclusion

In summary, the second brain method presented in the book follows the CODE system:

• Capture: Keep what resonates, leave the rest aside.

• Organize: Save for actionability (project, area, or resource).

• Distill: Find the essence of what a resource communicates.

• Express: Show your work based on the knowledge you gained.

Unless you’re a super memory genius, I strongly recommend looking into building a second brain. As we discussed, our brain should be for having ideas, not holding them.

Maybe the ideas in this post won’t fit you exactly, and you’ll come up with your own way of tracking notes. However you approach it, building a second brain has been incredibly important for me, and it might be for you as well.

📚 Resources

More From the Beyond the Code Category

• Don’t Forget About Your Mental Health

• Survivor Bias

• The XY Problem

Sources

• Building a Second Brain

Explore Further

• Create Your Own Second Brain

• Your Resource Guide to Building a Second Brain

❤️ If you enjoyed the post, please consider giving it a like. It’s a helpful signal to decide what to write next.

💬 Are you using a second brain system? If not, how do you keep track of your notes and ideas?

Leave a comment

Recently, I was going through the distributed systems reliability glossary brought by Antithesis & Jepsen, and I found their approach on availability models particularly interesting. Let’s dive into that.

Introduction

Here are two database whitepapers:

• Megastore, a Google storage system:

• Dynamo, an Amazon key-value store:

Both of these whitepapers use the term highly available. Of course, as a reader, we may expect these two whitepapers to mean the same thing when discussing high availability. After all, an orange is an orange, an apple is an apple, and a highly available system should be a highly available system. Yet, that’s not the case, and each paper means something different.

Let’s first look at what availability means, then discuss why high availability is a vague concept, and finally explore the different availability models.

Availability

In The CAP Theorem, we already discussed availability as:

Every request receives a non-error response, even if it may not contain the most up-to-date data.

However, this definition missed a crucial dimension: response time. Technically, a system could be available even if it responds after an hour. While such a system technically provides a non-error response, it fails to deliver a usable experience, severely compromising its practical availability from a user perspective.

This is where The PACELC Theorem offers a more practical perspective on availability. PACELC highlights that:

• In the presence of a partition, a system must choose between availability and consistency.

• In the absence of partition, a system must choose between latency (the upper-bound limit during which a request should receive a non-error response) and consistency.

So latency becomes part of availability. And that makes sense, right? If a system is too slow, it’s effectively down for the user. Availability isn’t just about uptime; it's also about whether the system is responsive in a meaningful timeframe.

High Availability

The term high availability is vague. Does it mean 99.9% uptime? 99.999%?

ScyllaDB, in their technical glossary says it’s about maintaining levels of uptime that exceed normal SLAs. That’s fine, but it’s also easy to game.

Say we define an SLA at 50%, then run at 80%. Technically, we exceeded it. Yet, does that mean we’re offering a highly available database? Probably not.

The Antithesis reliability glossary defines high availability as a system that is available more often than a single node. I quite like that one. If we take the availability of our best node and our system does worse than that, it’s not really highly available. Simple and practical.

Still, it’s not perfect. Let’s say we have five nodes, each available 50% of the time. With a write quorum of two, our write availability might still reach ~80%. But we’d still be down one request out of five. Hard to sell that as high availability.

To bring more clarity to the conversation, Antithesis introduced a set of availability models.

Availability Models

An availability model is something to help us define when an operation should succeed.

What do we mean by operation? It’s simply a request made to the system. That could be a read, a write, a ping, whatever the system is supposed to handle. Instead of thinking in terms of the whole system being up or down, we look at whether a specific request can succeed, even when parts of the system are failing.

Let’s explore three availability models.

Majority available

Definition: A system is majority available if when a majority of non-faulty nodes can communicate with one another, these nodes can execute some operations.

Consider a database composed of five nodes:

In the nominal case, everything works: a client connects, and the database can process all operations.

Now, imagine two nodes go down. Maybe they crash, maybe there’s a network issue causing a partition:

We’re left with three nodes that can still talk to each other. That’s a majority. If the database can still perform operations in this situation, we say it’s majority available.

That’s how the Megastore whitepaper defined highly available: being majority available.

This model is often used when consistency matters. For example, when a write or a leader election requires a majority to agree before responding to the client. So even if some nodes are unreachable, the system can still make safe progress as long as the majority is intact.

Total availability

Definition: A system is totally available if every non-faulty node can execute any operation.

Let’s take the same setup: a database with five nodes. This time, three of them are faulty:

In a majority available model, we can’t do much as we don’t have a majority. But in a totally available model, the system can still handle operations.

Indeed, in this model, each non-faulty node can act on its own. It doesn’t need to coordinate with others or wait for a network round-trip. This model favors latency. Just handle the request and move on.

That’s how the Dynamo whitepaper defined highly available: being totally available.

The tradeoff is consistency. Totally available systems can’t enforce strong guarantees because the nodes don’t necessarily sync before responding. That’s why this model typically goes hand in hand with weaker consistency models.

Sticky Available

Definition: A system is sticky available if whenever a client’s transactions are executed against a copy of database state that reflects all of the client’s prior operations, it eventually receives a response, even in the presence of indefinitely long partitions.

Let’s look at an example. Two clients, A and B, are connected to a database and make updates over time:

Sticky available means:

• After update 5, client A will eventually get a response that reflects at least updates 1, 4, and 5.

• After update 3, client B will eventually get a response that reflects at least updates 2 and 3.

How can we achieve that? It depends on how replication is handled by the system.

Fully Replicated System

In a fully replicated system, all nodes store the full dataset. Sticky availability can be achieved by making sure a client always talks to the same node.

Here’s the same example again, but now with client A always connected to node 1, and client B to node 2:

Here, node 2 hasn’t yet replicated update 5, and node 1 hasn’t yet replicated update 3. Still, since each client sticks to one node, they eventually see a consistent view of their own operations, despite possible failures such as a partition between the nodes.

Partially Replicated System

Now, let’s discuss a partially replicated system where nodes are replicas for subsets of data items.

Here’s a (dummy) partitioning system where even-numbered updates go to node 1 and odd-numbered ones to node 2:

Here, clients can’t just stick to a single node. Instead, they must maintain stickiness with a single logical copy of the database, which may consist of multiple nodes.

Clients can also help implement this model by acting as servers themselves. For example, a client could cache its own reads and writes, allowing it to return responses even during indefinitely long partitions.

Summary

• Highly available is too vague; watch out when you read or hear it. It might mean different things depending on the system or author.

• Majority available means a majority of nodes can still perform some operations. This model supports stronger consistency.

• Totally available means each non-faulty node can handle requests independently. It favors latency, but usually comes with weaker consistency.

• Sticky available means clients can make progress as long as they keep talking to a replica that reflects their own history.

• Availability models help us reason at the operation level, not just the system level. What matters is which operation can succeed, and under what conditions.

📚 Resources

More From the Distributed Systems Category

• The PACELC Theorem

• Exploring Database Isolation Levels

• Latency and User Experience

Sources

• A distributed systems reliability glossary - Antithesis

Explore Further

• High Availability Database Definition - ScyllaDB

• Consistency Models - Jepsen

❤️ If you enjoyed the post, please consider giving it a like. It’s a helpful signal to decide what to write next.

💬 When someone says their system is “highly available,” what do you assume they mean?

Leave a comment

Let me ask you a simple question: what makes a code beautiful?

Try thinking about this question (at least a little) before scrolling down.

You probably came up with a few characteristics: It’s readable and modular. Maybe it’s well tested. Or maybe it follows clean code principles or other familiar practices.

What if I told you that beautiful code is just… average code? That might sound silly, but let’s walk through it.

Attractive Faces Are Only Average is a foundational study by Judith H. Langlois and Lori A. Roggman in the psychology of facial attractiveness. They created composite images by averaging multiple individual faces—noses, eyes, cheekbones—all mathematically averaged.

These composites were consistently rated as more attractive than the individual faces used to make them. In fact, the more individual faces were averaged into a composite, the more attractive the resulting face became.

This suggests that facial features close to the population mean are generally preferred.

Why? The main theory is processing fluency: the easier something is for our brain to process, the more we like it. Average faces are easier to recognize and interpret, so they feel more pleasant.

This principle doesn’t stop at faces. It applies to advertising, art, or even decision-making. When something is easy to process, brain imaging studies show that it activates reward-related regions. We enjoy it more, without even realizing why.

So what happens when we apply this principle to code?

It means that the more average the code is, the more it aligns with what we’ve seen before, and the more beautiful it feels. The less surprising it is, the more pleasurable it is to read, understand, and maintain.

How can we write average code?

• Use standard naming conventions: Stick to familiar names so others instantly know what things are.

• Follow language idioms and common patterns: Write code the way others expect it to be written.

• Favor clarity over cleverness: The easier it is to read, the more enjoyable it is to work with.

• Be consistent: Predictability makes everything easier to understand.

• Minimize unnecessary novelty: The more our code looks like what we’ve seen before, the faster it gets processed and the more it’s liked.

Beautiful code is the kind that conforms to established patterns, idioms, and norms. It’s not flashy, novel, or overly clever. It’s clear, familiar, and quietly elegant. Just average.

What if next time, the best compliment your pull request could receive wasn’t that it was “brilliant” but simply “average”?

📚 Resources

More From the Software Engineering Category

• Cognitive Load: Reducing Mental Overhead in Software Design

• Readability: Understanding the What, the Why, and the How

• Premature Abstractions: Avoiding Unnecessary Complexity in Software Design

Sources

• Attractive Faces Are Only Average // The study mentioned in the post.

• Your Code as a Crime Scene // I read about this study from this book.

Explore Further

• Processing fluency

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 Does this idea resonate with you or does it feel wrong? Curious to hear your perspective.

Leave a comment

Last month, I was at the hospital with my partner for the birth of our newborn. During our stay, my partner experienced a specific symptom, and we wanted to understand what could be causing it. So we asked the midwife.

We immediately noticed the hesitation in her eyes. When she finally gave an answer, it came with a kind of forced confidence, and we both felt she wasn’t sure about it.

At our hospital, midwives do 12-hour shifts. So a few hours later, we asked the exact same question to the next midwife. Same hesitation, but this time, a different answer.

And so it went on. Every shift, we asked again. Every time, a different answer. Eventually, it even became a game between my partner and me: trying to guess what the next answer would be.

Until… The one. The one who broke this cycle.

We asked her the same question. She paused. Thought about it. And then said something unexpected:

It might seem counterintuitive, but these three words made us immediately trust her more than anyone else before1.

Twenty minutes later, she even came back to our room and said:

That brief exchange resonated with me.

In our field, we often put a lot of weight on posture. We build up our position as the go-to person for a codebase, a data model, or a framework. The more we know, the more we are seen as the one to consult or include in any related discussion.

But from that posture, admitting we don’t know something can feel like pulling out the bottom card in a house of cards. Suddenly, it feels like everything we built to earn that status might collapse.

Yet, if we take a step back, admitting we don’t know shouldn’t be seen as something shameful or embarrassing. In fact, it’s often the most responsible thing we can do.

• Pretending to know can lead to bad decisions. It creates false confidence and can steer a team in the wrong direction leading to possibly terrible outcomes.

• Authority isn’t built on knowing everything. It’s about being reliable and honest, someone others trust, and they can talk to with confidence.

• Teams work better when people feel safe to admit uncertainty. It fosters psychological safety among the team members, which Google identified as the top trait of effective teams.

• Curiosity + humility = real learning. Admitting we don’t know something is what keeps us learning and growing, no matter how experienced we are.

Whether it’s for us or others, next time we don’t know something, let’s be like that midwife: let’s just admit it. Without shame.

📚 Resources

More From the Beyond the Code Category

• Lateral Thinking

• Streetlight Effect

• 10 Rules I Learned About Technical Writing

Sources

• Understand team effectiveness

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 How comfortable are you with saying “I don’t know“?

Leave a comment

Introduction

When we build a system composed of multiple components (e.g., database, services, caches), it’s important to understand the dependency graph. For example, a service might depend on:

• A database to store data

• A messaging layer to exchange information

• A cache to reduce latency

Having a clear understanding of the dependencies in a system helps us maintain it more efficiently. But there's one question we often overlook: Are these dependencies soft or hard?

• Soft dependency: One that is non-critical for the service to operate properly.

• Hard dependency: One that is critical for the service to operate properly.

“Operate properly” in this context means, for example, that a service responds to requests, doesn’t lose data, and maintains an acceptable level of performance. In short, the service works reliably.

Two examples to illustrate the concept of soft and hard dependencies:

• A recommendation service is a soft dependency for a video platform. If it’s down, users can still watch videos, just without personalized suggestions.

• An authentication service is a hard dependency for a system that requires users to log in. If authentication is down, users can’t access the system.

Why It Matters

Understanding the type of dependency helps us make the right decisions:

• Reliability expectations:

  • Soft: High reliability expectation may not be necessary. Back to the example of a recommendation service for a streaming system, this service doesn’t need 5 9s availability (99.999%) if it isn’t on the critical user journey.

  • Hard: A hard dependency must match or even exceed the reliability of the dependent service. If a critical backend is only available 99.5% of the time but our own SLO is 99.9%, we have a structural problem. Setting the right expectation for a hard dependency is critical.

• Fault-tolerance strategy:

  • Soft: If the dependency is unavailable, we are not obliged to build a proper fault-tolerant strategy. We can let it degrade gracefully and wait for it to be back.

  • Hard: If the dependency is unavailable, we need to work on a strategy, such as establishing an efficient fallback strategy to keep our service running.

• Observability and alerting:

  • Soft: Observability is still important, but alerts can often have a lower priority or be routed differently.

  • Hard: The dependency must be tightly monitored. Failures or even minor degradation, such as latency spikes, error rates, or availability dips, must be tracked continuously.

• Rollout and change management:

  • Soft: Changes can be managed with more flexibility. Rollout may not require tight coordination or strict sequencing, and temporary failures might be acceptable.

  • Hard: Rollouts become delicate operations. We often need tight orchestration between teams, version compatibility checks, gradual rollouts with validation at each step, and well-tested rollback mechanisms. Any mistake could trigger a production incident.

Soft or Hard Dependency?

Classifying a dependency isn’t always obvious.

In some cases, it’s fairly straightforward. For example, if a REST endpoint requires a database query, that database is a hard dependency. But gray areas are fairly common, for example:

• A service can run without a certain dependency at runtime, but it still needs that dependency at startup to initialize. In this case, the dependency is hard from an operational point of view. If it’s down during a deploy or a scale-out, we can’t even get the service running.

• A service calls a soft dependency, but the RPC call has no timeout or fallback. If the dependency becomes unresponsive, the latency of our service spikes, possibly exhausting thread pools or request queues. What was supposed to be a soft dependency now puts the entire system at risk.

These are examples of soft dependencies not handled correctly, turning into hard ones in practice. Whether a dependency is technically optional doesn't matter if the failure of this dependency ends up blocking our service.

In many systems, identifying these cases is not trivial. Approaches like deliberately breaking dependencies or introducing hazardous conditions (e.g., random network delays) can help reveal which dependencies are truly non-critical and which ones only appear to be.

Evolutions Over Time

To make things even more complex, we need to keep in mind that the type of a dependency is not set in stone. A dependency that starts as soft can easily turn into a hard one over time.

Let’s consider a service that reads data from a database. We introduce a cache to reduce latency. Initially, this cache is a soft dependency. If it goes down, we fall back to the database, which results in an acceptable latency increase.

Yet, as traffic grows, the service begins to rely on the cache not just for latency but for throughput. At some point, if the cache becomes cold and every request hits the database, the database may no longer be able to handle the load.

In this example, the cache was a soft dependency, but it became a hard one due to changes in system conditions (more traffic).

This evolution (from soft to hard) is, unfortunately, much more common than the reverse. Without active effort on efficient maintenance and continuous testing, it’s fairly common for a soft dependency to turn silently into a hard one.

Improving Reliability

On the other hand, with active and continuous effort, it’s possible to turn a hard dependency into a soft one. One effective approach is to design a fallback strategy that makes the dependency’s downtime essentially invisible.

Designing a solid fallback is anything but simple (we’ll explore this in a future post). However one principle stands out: fallbacks need to be tested, and they need to be tested continuously. A fallback that hasn’t been exercised in months isn’t a fallback. It’s dead code.

Once we’ve reached a point where the dependency can go down and users don’t notice, then the dependency is soft. Turning hard dependencies into soft ones is one of the most effective ways to improve the reliability of a system.

Conclusion

• To manage dependencies effectively, we need to classify them as either soft or hard.

• To avoid surprises, we must understand that soft dependencies can turn hard without warning, especially as systems scale.

• To improve reliability, we should actively turn hard dependencies into soft ones using strategies like efficient fallbacks.

📚 Resources

More From the Reliability Category

• Reliability: The Most Important Feature a System Can Have

• Resilient, Fault-tolerant, Robust, or Reliable? The Key Differences Explained

• Graceful Degradation: Preventing Complete System Failures

Explore Further

• Defining SLOs for services with dependencies - Google Cloud

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 Have you seen a soft dependency quietly become critical over time?

Leave a comment

Why Track Mistakes?

Making mistakes is inevitable. Personally, I even wrote a book about mistakes, and, as I mentioned in it, I was a great source of inspiration for the content.

But what really matters is not making the same mistakes over and over. Ideally, we make a mistake once, extract a lesson, and ensure we never repeat it.

Do you know that making mistakes can actually help us grow? Research by cognitive scientist Janet Metcalfe suggests that reflecting on errors is one of the most effective ways to reinforce learning. Indeed, when we consciously analyze our mistakes, they don’t just fade away; instead, they reshape our thinking, helping us adapt and avoid repeating them in the future.

That’s why, for the past two years, I’ve been following a simple yet effective system to track, reflect on, and learn from my own mistakes. Of course, I don’t log every tiny one. Yet, my rule is simple: if a mistake makes me frustrated with myself, it’s worth tracking.

Entry Format

Each mistake I log follows a structured format:

• Name: A short, descriptive name.

• Tags: Relevant categories.

• Context: The situation in which the mistake happened.

• Problem: A description of the mistake itself.

• Impacts: The possible consequences of making this mistake.

• Lessons learned: What I can take away from it.

• Correction plan: What I will do to prevent this mistake in the future.

• Latest occurrence: When I last made this mistake.

• Repetition: How many times I made this mistake.

Here’s an example:

• Name: Skimming instead of reading carefully.

• Tags: Critical thinking, Attention to detail.

• Context: When reviewing an email, document, or technical spec.

• Problem: To save time, I sometimes skim through important content instead of reading it properly.

• Impacts:

  1. Leads to a partial understanding of a context.

  2. Causes misunderstandings, requiring additional clarifications.

  3. It can result in wrong decisions based on incomplete information, and/or I can look like an idiot.

• Correction plan: Better assessment of whether full attention is required. If it is, commit to reading properly.

Personally, I keep my mistake journal in Notion, but I guess any tool or even a physical journal could work.

Final Thoughts

Since implementing this system, I have noticed:

• Fewer repeated mistakes: I make the same mistakes less often.

• Pattern detection: It helped me recognize patterns in my own mistakes and better understand my own biases.

• Better decision-making: I catch potential mistakes earlier.

Mistakes are part of learning; they don’t have to be failures if we learn from them. By tracking them, we can turn them into lessons that help us grow and improve over time.

📚 Resources

More From the Beyond the Code Category

• Confirmation Bias

• Survivor Bias

• 10 Rules I Learned About Technical Writing

Sources

• Learning from Errors - Janet Metcalfe

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 Do you also track your mistakes? If yes, what’s your approach?

Leave a comment

Today, let’s dive into the world of testing and explore Property-Based Testing. In this post, we will discuss traditional tests, explore their limitations, and see how Property-Based Testing can help us improve our testing strategy by focusing on the fundamental concept of properties.

Traditional Tests

Imagine this white box contains all the behaviors implemented in a software:

Of course, like with any codebase, our software contains bugs (imagine how boring bugs-free software would be!). Let’s make the green part represent all the valid behaviors while the red part the invalid ones (the bugs):

Now, let’s write some tests to validate our code. For the sake of visualizations, we will partition the tests written into three sets: A, B, and C—all of them will be passing:

• Test set A: The true positive tests that correctly verify valid behaviors exhibited by the software. The remaining visible green part represents valid behaviors not covered by tests.

• Test set B: The false negative tests that should catch bugs, but they don’t because they are flawed. The remaining visible red part represents invalid behaviors not covered by tests.

• Test set C: The invalid tests that check for behaviors that… don’t even exist. They typically arise from misunderstandings of how a feature is supposed to work, and because the tests themselves are flawed, they fail to reveal this misalignment.

Let’s break down the four main problems in this example, sorted in increasing order of severity:

• 1. Invalid tests covering nonexistent behavior:

  

  These tests are fundamentally misaligned with reality. They cause unnecessary maintenance overhead and can mislead developers if tests serve as living documentation (see Unit Tests As Documentation).

• 2. Test coverage gaps:

  

  For now, this is okay. Yet, if something changes later in the uncovered area, regressions may go unnoticed.

• 3. Missing tests lead to undetected bugs:

  

  The consequences start to escalate. Here, bugs exist, but as there’s no test for them, they remain invisible.

• 4. Flawed tests hiding bugs:

  

  Even worse than 3., the tests here should detect bugs but they don’t. This is arguably worse than missing tests because it gives us a false sense of security: we think our software works as intended when it isn’t.

The Limitations of Traditional Tests

In this example, we rely on traditional tests. Regardless of whether we’re talking about unit tests, integration tests, or else, traditional tests follow a common structure:

• Define a starting state.

• Apply specific inputs.

• Assert that output matches expectations.

This method has been proven effective for decades, but it has limitations, including:

• Tests are manually designed, so some scenarios may be forgotten. Indeed, since traditional tests are written by humans, they are limited by our own assumptions and understanding of the software. Therefore, we can miss edge cases or unexpected situations.

• Passing tests don’t guarantee a bug-free system. A passing test only confirms that the tested scenario behaves as expected, not that the software works as expected. If the test is flawed, bugs can remain unnoticed.

• Test maintenance may become a burden over time. In contexts where high coverage is enforced, the more tests and edge cases we add, the more tightly coupled our tests become with the implementation, leading to increasing maintenance effort.

These limitations highlight a key challenge: traditional tests are only as good as the cases we manually define. They check specific scenarios but don’t ensure broader correctness across all possible inputs. This is where a different approach comes in, one that shifts the focus away from manually defining tests cases and instead focus on the fundamental concept of properties.

Property-Based Testing

Instead of defining specific inputs and expected outputs, Property-Based Testing (PBT) focuses on properties, meaning rules that must always hold, regardless of the input.

Rather than asking: “For this input, do I get this expected output?“ We’re asking: “Regardless of the input, does this property always hold?“

To give a concrete example, let’s discuss a common type of PBT tests known as fuzzing.

In general, fuzzing tests focus on the property: “My software shouldn’t crash“. For example, imagine we wrote a function that manipulates a string. Instead of manually testing edge cases, we can use a fuzzing library to generate random inputs (including unexpected formats or extreme values) to ensure our function doesn’t crash without requiring us to think of every possible edge case.

So, what’s the difference between fuzzing and property-based testing? To be honest, the boundary isn’t always crystal clear.

My working understanding is based on blog posts from Hypothesis, a property-based testing in Python (the post is referenced in the Sources section). In a nutshell, we should see fuzzing as a subset of PBT:

• Fuzzing: Primarily used to find crashes or unexpected behaviors by feeding random inputs. It doesn’t typically require a deep knowledge of the software’s expected properties.

• Property-based testing: Instead of just detecting crashes, PBT defines general properties that the software must always satisfy and generates test cases to verify them. This approach goes beyond failure detection and requires more structured thinking about correctness.

Both fuzzing and PBT start with randomized inputs (often called a fuzzer), but while fuzzing primarily focuses on crashes, PBT goes beyond that to formally validate expected system behavior.

So, what kind of properties can we define in property-based tests? Here are some examples:

• Structural properties: Ensure an operation preserves certain structural characteristics of the data, such as a length.

• Idempotency properties: Ensure that an idempotent function produces the same output when applied multiple times.

• Commutativity properties: Ensure that a function is commutative, meaning changing the order of its argument gives the same result.

• Roundtrip properties: For serialization or encoding functions, encoding followed by decoding should return the original value.

These are some classic properties we can think of in the context of PBT. Ultimately, PBT is about viewing code as a whole and identifying the fundamental properties it should always preserve. Instead of verifying isolated cases, we validate broader correctness guarantees:

Going Beyond

So far, the properties discussed in this post share similarities with unit tests, as they focus on local functions.

However, nothing in the concept of PBT prevents us from taking a step back and applying it at a broader level, such as at the API level or even at the system level, where multiple applications interact.

Once we extend this perspective, we can consider properties such as:

• Temporal properties: “99% of API calls must complete in less than 30ms“.

• Consistency properties: “The database must respect causal consistency”.

• Application invariants: “Financial transaction must be balanced (debits = credits)”.

• Reliability properties: “Retrying failed requests must not result in duplicate transactions”.

PBT is not limited to low-level function testing; it can be applied at multiple levels, from local functions to system level to assess property that must always hold for a group of applications.

With PBT, we can take any system, inject randomized inputs, and ensure that it behaves correctly—not by verifying specific test cases, but by enforcing fundamental system properties that must always hold, regardless of the input. This approach moves beyond individual test cases and shifts the focus to system-wide correctness.

Final Thoughts

PBT changes how we think about testing. Instead of manually defining scenario which may lead to issues like false negatives, PBT takes a step back and focuses on defining the fundamental properties or invariants that must always hold.

Beyond improving test coverage, PBT also serves as living documentation, as the properties capture essential rules that must govern our software or systems.

That said, PBT shouldn’t be seen as a replacement for traditional tests, it’s a complementary approach. The real challenge is recognizing that some parts of a system may be too complex to assess manually. In such cases, relying on randomized inputs and predefined properties can be a more effective and simpler way to catch bugs.

📚 Resources

More From the Testing Category

• Code Coverage

• Test Behavior, Not Implementation

• Avoiding Logic in Tests

Sources

• What is Property Based Testing? - Hypothesis Blog // The source I referenced in the post.

• In praise of property-based testing

Explore Further

• Software reliability, part 1: What is property-based testing? - Antithesis blog

• Property based testing: let your testing library work for you By Magda Stożek - Devoxx

• Proper and Basic Property-Based Testing

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 I purposely avoided discussing library-specific features because I wanted this post to encourage you to think about whether properties-based tests make sense for the systems you manage. Let me know what you think about PBT in the comments!

Leave a comment

📣 This post is part of a series written in collaboration with Antithesis, an autonomous testing platform. They are not sponsoring this post—I reached out to them because I was genuinely intrigued by what they were building and ended up falling in love with their solution. We will dive deeper into it in an upcoming post titled Deterministic Simulation Testing. In the meantime, feel free to check out their website or their great blog.

In tech, we often celebrate how much we accomplish: how fast we ship, how many hours we invest, how many open-source projects we juggle after work, etc. Yet, behind all these celebrations, something important often gets overlooked: our mental health.

We rarely talk openly about mental struggles or the exhaustion that may come from working in an industry that rewards constant hustle.

This post is a reminder (for all of us, including me) that taking care of our minds isn’t optional: it’s a critical part of the process to keep growing over time.

Manage Your Energy, Not Your Time

Time is constant1. We all get the same 24 hours, day after day. Yet, have you noticed how some days feel wildly more productive than others, even when our schedule looks exactly the same?

That difference usually isn’t about having more time. It’s about having more energy, especially mental energy.

We tend to think burnout happens when we run out of time. But burnout really sets in when we run out of energy and keep pushing anyway, when we ignore what our mind and body are telling us just because our calendar says we’re “free“.

If our energy is low, our mental health is trying to tell us something, and we should listen.

Managing our energy instead of our time means respecting our limits and acknowledging that we are not machines. For example, there are days when my energy is low, and through experience, I’ve learned it’s usually far better to take care of myself first rather than forcing myself to write another post. Do whatever helps you refill your energy: rest, take a walk, play video games, whatever works for you.

It’s not laziness. A career is a marathon, not a sprint. Burnout happens when we push ourselves too hard, too often. Thinking long-term means paying attention to our energy because it’s the best indicator we have for how sustainable our pace really is.

Don’t Compare Yourself to Others

One of the quickest ways to harm our mental health is by constantly comparing ourselves to others. When we keep looking at what everyone else is doing, we start to question if we’re doing enough.

A few years ago, I saw a former classmate working as a senior software engineer at Netflix. At the time, it made me feel like I was falling behind. But looking back, I realize I was just being too hard on myself.

If you've ever felt the same, remind yourself of this: you're not behind, you're just moving at your own pace.

Comparing ourselves to others becomes even more harmful on social media. It's easy to scroll through and see people who seem more successful than we are. But the truth is, social media rarely shows the full story. We don't see their struggles, rejections, or failures. We're mostly seeing the bright side.

If scrolling leaves us feeling anxious or not good enough, it’s totally okay to take a break. Stepping back helps us refocus on ourselves and reminds us what truly matters: our own journey at our own pace.

It’s Okay to Struggle Sometimes

Sometimes, we can go through tough periods, not just bad days, but weeks or even months. We might feel anxious, unmotivated, disconnected. It can be hard to focus, hard to care, hard to show up. That doesn’t mean we’re weak; it means we’re human.

We don’t have to be perfect all the time. We don’t have to hide our struggles or pretend everything is fine. Life has ups and downs, and it’s okay to move through these difficult periods without guilt, regardless of the reasons.

When we're struggling, one of the best things we can do is talk to someone: a friend, a partner, a mental health professional, whoever. There's always someone who can help us carry the load. Just sharing what we're feeling out loud can lighten the emotional weight a little.

And if we're in an company where we can't openly talk about mental health, where we can't admit we're struggling without fear or judgment then that's a company issue, not ours. If our working environment doesn't support our well-being, it’s probably not the right place for us.

Everyone goes through hard times. What matters most is giving ourselves permission to feel, heal, and reach out when we need support.

Final Thoughts

Mental health isn’t something separate from our work; it’s at the heart of it. Managing our energy, avoiding comparison, and accepting that it's normal to struggle are essential for a healthy, sustainable career.

Remember: we are humans first, engineers second. Taking care of ourselves isn’t just good for us; it's good for everything and everyone around us.

Let’s keep building thoughtfully, sustainably, and kindly, together.

📚 Resources

More From the Beyond the Code Category

• Cognitive Biases

• Keeping a Mistake Journal

• The XY Problem

Sources

• How to manage mental health in the tech industry

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 I would genuinely love to hear your thoughts on this post. How do you take care of yourself when things get hard?

Leave a comment

Introduction

Throughout my career, I’ve worked in many complicated environments. For instance, I worked on optimizing driver-passenger matching in ride-hailing at a Uber's competitor. This context, like others, was technically challenging. Yet, nothing comes close in terms of complexity with my current experience at Google and two years there have refined my perception of complexity.

In this post, we will break down the very concept of complexity. Next, we will take a step back to understand what makes certain environments rather complex than complicated and then explore patterns for navigating complex systems effectively.

Complicated vs. Complex

Understanding the distinction between complicated and complex problems is crucial because each requires a fundamentally different approach:

• Complicated problems are intricate but predictable. They followed structured, repeatable solutions. For example, filing taxes is complicated, but it's a structured and conventional problem since the process remains mostly the same year after year.

• Complex problems are intricate but unique. They require adaptive and often novel solutions. For example, climate change mitigation is a complex problem because it requires new and adaptive solutions and existing methods alone can’t really address its evolving challenges.

Back to software engineering, at the Uber competitor, one of the challenges was to efficiently find the nearest driver for a passenger. This was far from trivial, but it wasn't complex per se. Indeed, many solutions exist, such as applying geo-hashing (example), and implementing one of these solutions was the right approach.

At Google, I work as a Site Reliability Engineer (SRE), focusing on the systems powering Google’s ML infrastructure. Here, I consider the challenges genuinely complex, as new paradigms and scheduling approaches are required, especially at Google’s scale.

Recognizing whether a system is complicated or complex is really important. Indeed, we mentioned that complicated systems are by definition repeatable, while complex systems require unique and customized approaches. Therefore, if we try to apply a common solution to a complex problem, it may not lead to effective results.

Characteristics of Complex Systems

In this section, we will discuss five common characteristics that help identify complex systems. Not all complex systems share every characteristic, but they tend to exhibit at least some of the following.

Emergent Behavior

Emergent behavior arises when a system's overall behavior cannot be predicted solely by analyzing its individual components in isolation.

For example, Gemini producing unexpected results was an emergent behavior. While I can't disclose the root cause, this behavior was nearly impossible to foresee by analyzing all the different components separately.

This is one possible characteristic of complex systems: they behave in ways that can hardly be predicted just by looking at their parts, making them harder to debug and manage.

Delayed Consequences

Another possible characteristic of complex systems is delayed consequences, where actions don't always have immediate effects, and instead, consequences may only become apparent much later.

For example, deploying a new version of a system might introduce a subtle issue that only appears days or even weeks later. This delay complicates debugging since identifying the root cause becomes much harder compared to immediate impacts.

In complex systems, relying solely on immediate feedback can create a false sense of stability, leading to major surprises when an issue finally emerges. Keeping in mind that consequences may take time to surface is essential when working in such environments.

Local vs. Global Optimization

In complex systems, optimizing one part doesn’t necessarily improve the whole system, and in some cases, it can even make things worse.

Unlike in non-complex systems, where improving one part generally leads to positive gains, complex systems are much more difficult to reason about. The components interact in non-obvious ways, and local optimizations can create ripple effects that are difficult to predict, sometimes leading to negative outcomes at the system level.

This highlights a key trait of complex systems: the whole is more than the sum of its parts. As a result, local gains don’t always translate into global improvements and in some cases, they can even degrade the overall system.

Hysteresis

Hysteresis describes how a system's past state continues to influence its behavior, even after the original cause is removed.

A real-world example to illustrate hysteresis is traffic congestion: even after a road accident is cleared, delays persist because vehicles remain clustered. Similarly, in distributed systems, failures can cause cascading slowdowns, even after the root issue is fixed. Indeed, dependent systems may take time to recover for various reasons, such as caches, retries, or queued requests.

In complex systems, simply fixing the root cause is not always enough. Therefore, it's crucial to assess whether a system is prone to hysteresis and, if so, anticipate its effects.

Nonlinearity

In complex systems, small changes can produce disproportionately large or unpredictable effects.

For example, in queueing theory, system load increases latency predictably. However, as a queue approaches saturation, even a small increase in requests can cause response times to spike exponentially.

Complex systems often reach tipping points, where behaviors shift suddenly, making past trends unreliable for prediction. This nonlinearity means that traditional linear assumptions where inputs map predictably to outputs isn't always effective for designing, testing, and reasoning about complex systems.

Summary

To summarize this section, complex systems:

• Are difficult to understand just by looking at its parts separately.

• Don’t always show their effects right away, consequences can be delayed.

• Don’t always improve as a whole when one part is optimized and changes can sometimes make things worse.

• Can keep being influenced by past states, even after the original cause is gone.

• Can react to small changes with big or unexpected effects.

Note that scale alone doesn’t make a system complex: even small systems can exhibit complex behaviors like emergence or nonlinearity.

Patterns for Navigating Complex Systems

Given these characteristics, how can we operate effectively in complex environments? Below are some strategies that I personally found effective.

Reversibility

When dealing with complex systems, we should favor reversible decisions whenever possible, meaning changes that can be undone if they don't work out.

Amazon's one-way vs. two-way doors framework captures this idea quite well:

• One-way doors represent irreversible decisions that require careful deliberation.

• Two-way doors represent reversible decisions, allowing us to move fast and iterate with lower risk.

In many contexts, especially in complex systems, favoring two-way doors leads to better outcomes because we can experiment, learn, and refine rather than overengineering upfront.

That being said, not all decisions should be reversible. For example, some choices like security policies or compliance-related changes require upfront commitment. The key is knowing when to optimize for speed and iteration versus when to be deliberate and careful.

Think Beyond Immediate Metrics

Because complex systems don't always respond predictably to local optimizations, defining the right metrics for success is probably just as important as the changes we make. Indeed, focusing too much on isolated, local metrics can create a false sense of success while masking unintended negative consequences elsewhere in the system.

To avoid this, before making a change, we should define both local and global metrics to get a holistic view of system health. This ensures that we measure impact beyond the immediate area of focus and consider the system as a whole.

Well-chosen metrics shouldn't just confirm the success of a local change; instead, they should help us make better decisions and ensure meaningful improvements at the system level, not just isolated areas.

Innovation

As discussed, complex systems often demand unique solutions. Since conventional strategies may not always apply, we must be willing to think out of the box and embrace innovation.

I recall one of my first meetings at Google. Someone presented a problem that seemed absurd in terms of complexity, especially given the scale. My immediate reaction in my head was: "This is impossible". But then, a teammate said: "But we're Google, we should be able to manage it!".

That remark stuck with me. While not every company obviously has Google's resources, I think the mindset is what truly matters. When facing a complex problem, we should assume it's solvable, then break it down, experiment, and iterate until we find a path forward.

One may find this section cliché, but again, complex problems demand unconventional thinking. In many cases, being open to innovative solutions when facing a complex problem isn’t just helpful, it’s necessary.

Controlled Rollout

When deploying changes in complex systems, we should rely on proven best practices to minimize risk. These include:

• Feature flags: Enable or disable functionality dynamically without deploying new code, allowing for safe experimentation and quicker rollbacks.

• Canary release: A limited rollout to a small, controlled subset of production, well suited for environments with only a few production instances.

• Progressive rollouts: Gradually increasing the scope of a rollout, best suited for large-scale production setups with multiple clusters or regions.

• Shadow testing: Running a change in parallel with production traffic without impacting real users. This helps validate the correctness of a change before enabling it.

By leveraging these techniques, we reduce the blast radius of failures, improving the confidence in our changes and enabling faster iterations.

Observability

Observability is one of the main pillars of complex systems. My working definition of observability (mainly inspired by Observability Engineering) is the following:

You can understand any state of your system (no matter how novel or bizarre) by slicing and dicing high-cardinality and high-dimensionality telemetry data without needing to ship new code.

Without observability:

• Systems become more fragile as unknown issues remain hidden until they cause real impacts.

• Debugging unexpected failures becomes significantly harder.

• Innovation is slowed down due to a lack of efficient feedback loops.

In complex environments, where unknowns are inevitable, observability is essential. It enable teams to navigate uncertainty, experiment more safely and getting short feedback loops to continuously improve systems.

Without proper observability, changes remain opinions rather than informed decisions.

Simulation

Predicting the behavior of complex systems is rarely simple, and, sometimes, nearly impossible.

I recall a case where we spent considerable time designing a change, carefully backing every assumption with data. Yet, due to unaccounted factors such as lurking variables, the change was ultimately ineffective1.

Sometimes, instead of relying solely on predictions, a more effective approach can be to simulate a change before rolling it out. There are multiple ways to leverage simulation testing, including:

• Replaying past events: If we design a system to record all its input, we can replay past events against our new version and analyze its impact. This allows us to validate changes in a controlled manner, reducing uncertainty and improving decision-making in complex systems.

• Deterministic simulation testing: Instead of relying on real-world data, we can create controlled, repeatable simulations that model system behavior under specific conditions. This allows us to test how a system reacts under various conditions in a fully deterministic way.

Note that the ideas presented in this section also rely heavily on observability.

Machine Learning

In complex environments, rules-based approaches often reach their limit because of the complexity of anticipating all scenarios. In these contexts, ML can become particularly effective.

Indeed, unlike static heuristics, ML models can continuously adapt based on feedback loops and learn from real-world data rather than relying on rigid, predefined logic.

This allows systems to:

• Detect emerging patterns that weren't explicitly programmed.

• Adapt dynamically to changes without requiring constant human intervention.

• Make probabilistic decisions rather than relying on strict if-else conditions.

Strong Team Collaboration

Last but not least, I believe that in complex environments, more than anywhere else, strong team collaboration is an absolute necessity. For instance, clearly conveying why a change is complex, discussing available options, and debating trade-offs with teammates are all critical skills.

In complex systems, there's often no single right answer. Therefore, a team that collaborates effectively and navigates ambiguity together can make a huge difference, ultimately leading to stronger decision-making.

Final Thoughts

Again, complicated problems can be solved with repeatable solutions, whereas complex systems require adaptability and a different way of thinking. This is why recognizing whether a system is complicated or complex is so important: it shapes how we should approach problem-solving.

However, in many environments, systems are neither purely complicated nor purely complex. Some parts can follow structured, predictable solutions, while others require adaptability and novel approaches. The key is learning to recognize when adaptability is needed and when structured solutions are enough.

📚 Resources

More From the Distributed Systems Category

• Latency and User Experience

• Isolation Level

• Probabilistic Increment

Sources

• The Critical Difference Between Complex and Complicated

• Jeff Bezos explains one-way door decisions and two-way door decisions:

Explore Further

• Hacker News discussion

• Simple, Complicated, Complex, and Chaotic Systems

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 I hope this post has helped you recognize the characteristics of complex environments and provided you with practical patterns to navigate them effectively. Did any of these patterns resonate with you? What other strategies have you used in complex environments? Let me know in the comments.

Leave a comment

📣 This post is part of a series written in collaboration with Antithesis, an autonomous testing platform. They are not sponsoring this post—I reached out to them because I was genuinely intrigued by what they were building and ended up falling in love with their solution. We will dive deeper into their solutions in a future post. In the meantime, feel free to check out their website or their great blog.

Understanding the Max Flow Problem

The maximum flow problem involves finding the highest possible flow that can travel from a source to a sink in a flow network without exceeding edge capacities.

Key terminology:

• Flow: The amount of “stuff“ (e.g., water, data, cars) passing through an edge without exceeding the edge’s capacity.

• Capacity: The maximum amount of flow an edge can handle.

• Vertex: A node in the network where the flow can enter or exit.

• Edge: A directed connection between two vertices that carries flow and has a capacity limit.

• Source: The starting point of the flow, where all the "stuff" originates.

• Sink: The endpoint where all the "stuff" is collected.

Some real-world applications:

• Water pipeline system: A network of pipes where water flows from a reservoir to homes with pipes having different capacities. What’s the maximum water flow that can be transported?

• Data networks: Internet packets traveling through routers, where each connection has a bandwidth limit. What’s the maximum throughput between two routers?

• Traffic flow: Routes connecting cities, where cars move from an entry point to an exit, and each route has a capacity limit. What’s the maximum number of cars that can travel between two cities?

Augmenting Path

Before diving into the algorithm, we first need to define the concept of augmenting path:

• A path from the source to the sink is a sequence of connected vertices that begins at the source and ends at the sink.

• An augmenting path is a path from the source to the sink in which every edge has available capacity for additional flow.

For example:

• A → B → D is both a valid path and an augmenting path because all edges along the path have remaining capacity.

• A → C → D is a valid path but not an augmenting path because C → D has no remaining capacity, meaning no additional flow can pass through this path.

Ford-Fulkerson Algorithm

Consider the following graph where we need to find the maximum flow from A to F:

Intuition Behind the Algorithm

One possible intuition for solving this problem could be:

1. Find an augmenting path from A to F (i.e., a valid path where all the edges have remaining capacity).

2. Calculate the bottleneck capacity—the maximum flow that this path can carry, limited by the edge with the smallest available capacity.

3. Update the capacity of the edges along this path by subtracting the bottleneck flow.

4. Repeat steps 1-3 until no more augmenting paths exist.

Let’s follow this intuition and apply it step by step.

Finding Augmenting Paths

Note that the Ford-Fulkerson algorithm doesn’t impose a specific way to find augmenting paths. For this example, we use Depth-First Search (DFS).

Suppose the first augmenting path found is A → B → E → F:

The bottleneck for this path is 2 as any additional flow would exceed A → B capacity.

We push a flow of 2 through this path. Then, to reflect on the flow that has been sent, we update the remaining capacity of all edges in this path by removing 2:

Capacity updates:

• A → B: The capacity went from 2 to 0 (-2), so we removed the edge. We could also keep a capacity of 0; it doesn’t really matter.

• B → E: The capacity went from 3 to 1 (-2).

• E → F: The capacity went from 4 to 2 (-2).

Now, let’s find another augmenting path using the same technique, DFS. Suppose the next path found is A → C → E → F:

The bottleneck for this path is 2 (bounded by E → F). So again, we update the remaining capacity of all the edges in this path:

Capacity updates:

• A → C: The capacity went from 8 to 6 (-2).

• C → E: The capacity went from 6 to 4 (-2).

• E → F: The capacity went from 2 to 0 (-2), so we removed the edge.

A Correct Solution?

At this stage, there is no more augmenting path left from A to F. Let’s summarize:

• We found a first augmenting path A → B → E → F with a max flow of 2.

• We found a second augmenting path A → C → E → F with a max flow of 2.

Therefore, if our intuition was correct, the maximum flow between A and F should be 4. However, this isn’t the right solution. Instead, we should have considered these two augmenting paths:

• A → B → D → F: 2

• A → C → E → F: 4

If we had taken these two augmenting paths, we would have found the correct solution: 6.

What Went Wrong?

Should we refine our way of selecting augmenting paths? Not with Ford-Fulkerson. As we said, augmented paths in Ford-Fulkerson can be selected arbitrarily, so let’s keep using DFS.

Instead, we’re missing a crucial step: whenever we select an augmented path and send flow through an edge, a backward edge with the same flow value must be created.

The purpose of this step is to allow flow redistribution, meaning if an unlucky selection of augmenting paths leads to a suboptimal result, the algorithm can undo and redistribute the flow when needed. Let’s see how that works.

Incorporating Backward Edges

We restart the algorithm to find the maximum flow from A to F, this time properly incorporating backward edges.

The first augmented path found is still A → B → D → F:

Now, instead of just adapting the capacity, we will create a backward edge based on the flow sent. In this case, the flow sent is 2 so we will create three backward edges with a capacity of 2:

Capacity updates:

• A → B: The capacity went from 2 to 0 (-2), so we removed the edge.

• B → E: The capacity went from 3 to 1 (-2).

• E → F: The capacity went from 4 to 2 (-2).

Meanwhile, we added three backward edges with a capacity based on the flow sent (2):

• B → A: 2

• E → B: 2

• F → E: 2

Why we’re doing this may not be clear yet, but let’s keep going—you will soon see why it makes sense.

Now, let’s consider the second path, A → C → E → F:

The max flow for this path is 2. We follow the same steps by updating the capacity for all the edges in the path (-2) and all the backward edges (+2).

We update all the capacity and create 2 new backward edges:

• C → A: 2

• E → C: 2

The F → E backward edge was already present from the first augmented path, so we increment its capacity: 2 + 2 = 4.

At this stage, we are at the same point as before, where we have found two augmenting paths with a flow of 2. However, there’s now a major difference: thanks to backward edges, there is one more remaining augmenting path: A → C → E → B → D → F. Note that this new path uses a backward edge that we created: E → B. Let’s consider this path:

The path’s bottleneck is 2 (bounded by E → B). Let’s adapt the capacities:

• -2 for all the edges in the path (including E → F).

• +2 for all the backward edges.

Note that as we decreased by 2 the capacity of E → B, it reached a capacity of 0, so we removed the edge.

Is there a remaining edge? No, that’s the termination condition of Ford-Fulkerson. Here are all the augmenting paths that we found and their corresponding flows:

• A → B → E → F: 2

• A → C → E → F: 2

• A → C → E → B → D → F: 2

If we sum all the flows, we get 6, which is the correct maximum flow.

This is how the Ford-Fulkerson algorithm guarantees that, even with arbitrary augmenting path selection, it will always find the correct maximum flow thanks to backward edges allowing flow redistribution.

Code

• Pseudo-code:

  def ford_fulkerson(graph, source, sink):
      res = 0
      while True:
          path = get_next_augmenting_path(graph, source, sink)
          if path is None:
              break
          
          bottleneck = get_bottleneck(path)
          for edge in path.get_edges():
              edge.capacity -= bottleneck
  
              backward_edge = get_backward_edge(graph, edge)
              backward_edge.capacity += bottleneck
          
          res += bottleneck
      
      return res

• You can also look at my Go solution.

Complexity

The time complexity of Ford-Fulkerson algorithm is O(EU), where:

• E is the number of edges in the graph.

• U is the maximum possible flow (i.e., the solution).

It’s quite interesting to note that the algorithm’s complexity depends on the final solution itself. This is something fairly unusual because, ideally, we don’t want to solve the problem first just to determine its time complexity.

We can, however, approximate U, as it depends on the edge capacities. If we increase the edge capacities, then in most cases, the maximum flow U will also increase. Yet, this isn’t the only factor and we should also consider:

• Bottlenecks: If a single edge has low capacity, it limits U no matter how large other edges are.

• Graph structure: Even if edges have large capacities, if there are few paths from source to sink, U may not be very large.

• Parallel paths: If there are many disjoint paths from the source to the sink, U will be higher even if individual edges have small capacities.

Yet, one important question: What if we need an algorithm whose complexity doesn’t depend on U since U isn’t straightforward to estimate?

Amazingly, the solution is to simply change how we determine augmenting paths. In our example, we used DFS, which led to an O(EU) complexity. However, if we switch Breadth-First Search (BFS) instead, the time complexity becomes O(VE²), where:

• V is the number of vertices (nodes) in the graph.

• E is the number of edges in the graph.

Said differently, with BFS, the time complexity depends only on the graph structure (vertices and edges). This removes the difficulty of predicting an accurate value of U.

This specialization of Ford-Fulkerson using BFS is known as the Edmonds-Karp algorithm.

Summary

• Ford-Fulkerson repeatedly finds augmenting paths and pushes flow through them.

• Backward edges allow redistribution of flow, which corrects any suboptimal choices made by selecting arbitrary augmenting paths.

• The algorithm terminates when no more augmenting paths exist.

• Using DFS, the time complexity is O(EU), which depends on the final solution, but switching to BFS (Edmonds-Karp variation) results in a time complexity of O(VE²), making it dependent only on the graph structure.

📚 Resources

More From the Coding Category

• Topological Sort

• Arrays vs. Linked Lists

• Binary Heaps

Sources

• Maximum flow problem - Ford Fulkerson algorithm // The graph example I used in this post is taken from this video. If you struggle with my explanation, this video may help you.

Explore Further

• Dinic’s Algorithm - YouTube // An explanation of Dinic’s network flow algorithm, another way to solve max flow problems.

🎉 Solution

The correct solution to the 1,000 Subscribers, 1 Coding Challenge! was 56. Congratulations to everyone who solved it! The winners (Simon, María, Alexandr, Brian, Vadim), chosen randomly from all correct submissions, will be notified by email.

Thanks to all who participated! I hope you enjoyed the challenge. Let me know if you'd like to see more of these in the future!

❤️ If you made it this far and enjoyed the post, please consider giving it a like.