Stable, Vulnerable, Metastable

Metastable failures borrow their name from physics, where metastable means something that looks stable but isn’t. To understand how a distributed system can end up in such a state, we need to look at three distinct states it can be in:

• Stable: The system recovers on its own after any disruption. This is what we call resilience in Resilient, Fault-tolerant, Robust, or Reliable.

• Vulnerable: The system looks perfectly healthy, but it's operating above its hidden capacity: the load level below which it can self-heal from any disruption. It responds fast, metrics are green, and nothing is alarming. Many production systems deliberately operate here because it's more efficient: resources are used closer to their limit. But there's no slack left. And the deeper the system operates in a vulnerable state, the smaller the trigger needed to push it over the edge. Indeed, a system just above its hidden capacity can survive large disruptions; a system near its advertised capacity can be tipped by almost anything.

• Metastable failure: A trigger (e.g., a network blip, a deployment, a traffic spike) pushes the system over its hidden capacity. The system is not fully broken: processes are alive, and it’s still running. But goodput collapses: it’s no longer doing any useful work. Technically up, effectively down. And unlike a regular outage, removing the trigger doesn’t fix it. Getting out requires a strong corrective push: a restart, a dramatic load reduction, a manual intervention.

  NOTE: If you’re not familiar with the concept of goodput, it’s the throughput of useful work completed successfully. For example, in a web application receiving 1000 requests per second but returning errors for 800 of them, the goodput is only 200 RPS.

The most disorienting property of a metastable failure: stopping the trigger doesn’t stop the failure. To understand why, we need to talk about feedback loops.

The Feedback Loop is The Root Cause

In a previous post on Systems Thinking Explained, we defined a feedback loop as: If A causes B, then B influences A.

A feedback loop is exactly the mechanism that keeps a system stuck in the metastable state. There is always a sustaining effect, a feedback loop, that prevents recovery. The trigger is just what pushes the system over the edge. The loop is what keeps it there. Blaming the trigger is the natural instinct, and almost always the wrong diagnosis.

Let’s discuss a concrete example to make this clear.

Imagine a web application that queries a database. The database comfortably handles up to 300 QPS. The application retries any query that doesn’t respond within 1 second. The system is running at 280 QPS, healthy and fast, within the database’s capacity.

Then, a transient network issue occurs for 10 seconds. When the issue is over, all the queued requests flood in at once. The database gets hit with a surge it can’t absorb: latency spikes and queries start timing out. So the application retries them. This doubles the effective load to 560 QPS. The database, already struggling, falls further behind. More timeouts. More retries. The loop is now self-sustaining:

High load → Timeouts → Retries → Higher load → More timeouts → More retries

The transient network issue was fixed minutes ago. Yet, the system is still completely broken. The trigger is gone; the feedback loop is not. The only way out is to dramatically cut the load or disable retries entirely.

This is a metastable failure. The system was vulnerable because it was operating close to its hidden capacity. A minor, transient trigger pushed it over the edge and into a self-sustaining failure state it couldn’t escape on its own. The retry mechanism, a feature designed to improve reliability, became the very thing that prevented recovery.

This is one example, but the same pattern appears with caches, connection pools, failover logic, and more. The shape is always the same: a feedback loop that turns a temporary problem into a permanent one.

Why It’s So Hard To Fix

Two things make metastable failures particularly nasty.

• We can be tempted to blame the wrong thing. When an outage happens, the trigger is what’s visible and recent: a spike, a deployment, a hardware fault. It’s the obvious culprit. But the trigger only exposed the problem; it didn’t create it. The sustaining feedback loop was already there, structural and invisible. When analyzing the problem in retrospect, teams focus on the trigger; fixes address the trigger; and the system remains vulnerable to the next one. The authors of the paper observed teams declare a metastable failure “resolved” multiple times before realizing the real cause had never been touched.

• The feedback loop grows stronger with scale. Small-scale tests won’t reveal it. A staging environment running at 10% capacity may handle the same trigger without falling into a metastable state, because the loop isn’t strong enough at that scale to be self-sustaining. This means these failures can slip past even rigorous testing regimes and only manifest in production at full load.

We defined hidden capacity earlier as the load level below which the system can self-heal from any disruption. It’s different, and always lower, than the advertised capacity. In our example, the numbers make it concrete: the advertised capacity is 300 QPS, but the hidden capacity is only 150 QPS, because retries double the load under failure. The gap between those two numbers is where vulnerability lives.

Measuring the hidden capacity is not straightforward, though. One possible approach is to apply a trigger at a given load level and observe whether the system recovers on its own:

• If it does, we are below the hidden capacity.

• If it doesn’t, we are above it.

We can also estimate it indirectly: in the retry example, retries double the load under failure, so the hidden capacity is roughly half the advertised capacity.

Finding the Loop, Not the Bug

Metastable failures are not bugs. We can’t write a unit test that catches them. They are emergent behaviors: properties that arise from the interaction of a system’s components under specific conditions, not logic errors in any individual component. No single piece of code is buggy, no single configuration is wrong. The failure is a consequence of how everything fits together under load.

This changes how we need to think about them. The right question after an outage is not “What failed?” but “What loop sustained it?” And before an outage, the danger is not having bugs; it’s optimizing so aggressively for efficiency that we push the system deeper into the vulnerable state without realizing it.

Retries, caches, failover logic, connection pools: these are all features that improve reliability in the common case. They are also, under the right conditions, the sustaining mechanisms of metastable failures. The same design decision that makes a system more resilient in normal operation can also prevent it from recovering when things go wrong.

What Can We Do About It?

The paper describes several approaches to reduce the risk of metastable failures:

• Retry budgets and circuit breakers: Instead of retrying indefinitely, cap the total number of retries in flight at any given time. This directly weakens the feedback loop by limiting work amplification.

• LIFO scheduling under overload: Counterintuitively, switching from FIFO to LIFO when the system is overloaded allows some requests to complete within their deadline, preserving goodput instead of letting every request time out.

  NOTE: I already wrote a post about that approach in Adaptive LIFO.

• Fast error paths: Success paths are heavily optimized, but error paths often aren’t. An expensive error path (stack traces, DNS lookups, disk writes) under high failure rates can itself become a sustaining mechanism. Optimizing error paths reduces this risk.

• Read-through caches over look-aside caches: A read-through cache (where the cache itself fetches missing data from the database) can continue filling itself even when the application has given up on a request, steadily increasing the hit rate and helping the system recover. A look-aside cache (where the application is responsible for populating the cache) can’t.

• Production stress testing: Small-scale tests won’t reveal metastable failures. Testing against a portion of production traffic, with engineers ready to intervene, is the most reliable way to surface them.

A note of humility from the paper: there is no systematic solution yet. These are ad-hoc mitigations developed in response to known failures. Detecting vulnerable states before they collapse remains an open problem.

Summary

• A distributed system can pass through three states: stable, vulnerable, and metastable. The vulnerable state looks healthy, but it isn’t.

• The threshold between stable and vulnerable is invisible. Systems can operate in the vulnerable state for months without any sign of trouble.

• When a trigger pushes a vulnerable system into a metastable failure, a feedback loop sustains the failure even after the trigger is gone.

• The trigger is not the root cause. The feedback loop is. Fixing the trigger leaves the system vulnerable to the next one.

• Reliability features like retries and caches can become the sustaining mechanism of a metastable failure under the right conditions.

• Metastable failures are emergent behaviors, not bugs. We can’t unit test for them, and optimizing for efficiency makes them more likely.

• Mitigations exist (retry budgets, circuit breakers, LIFO scheduling, fast error paths), but they are all ad-hoc responses to known failures. Detecting vulnerable states before they collapse remains an open problem.

Resources

More From the Reliability Category

• Resilient, Fault-tolerant, Robust, or Reliable?

• Adaptive LIFO

• Fail Open vs. Fail Closed

Sources

• Metastable Failures in Distributed Systems

Explore Further

• Metastability and Distributed Systems

Fundamental Insights

To understand LSM trees, we first need to understand why writes are hard.

A B-tree-based database updates data in place. When we write a key, the engine finds the right page on disk and modifies it. This is a random write: the disk head has to seek to an arbitrary location before writing. On spinning disks, that seek takes time. But even on SSDs, random writes cause problems: they wear out cells unevenly and trigger expensive internal garbage collection.

LSM trees take a completely different approach. Instead of writing data where it ultimately belongs, they write data sequentially. Writes are recorded in memory and appended to a log file for durability. When the in-memory buffer fills up, its contents are streamed to a new file in one sequential pass. Sequential writes are dramatically faster than random writes because there is no seeking involved. The disk just keeps writing forward.

The price of this design is complexity. Data doesn’t live in one place. It accumulates across multiple files over time, and those files need to be periodically merged and reorganized in the background to stay manageable. That background work is what every piece of an LSM tree is built around.

The in-memory buffer is called the memtable. The sorted files on disk are called SSTables. We’ll look at each in detail.

The Memtable

Every write in an LSM tree starts in memory, in a structure called the memtable.

The memtable is a mutable, in-memory store. When a write request arrives, the engine records the key-value pair in the memtable and appends it to a sequential log file on disk (called the write-ahead log, or WAL, which we’ll cover in the next section). The WAL write is a sequential append, so it is fast. There is no random I/O, no page lookup, no in-place modification. This is why LSM trees can sustain very high write throughput.

A hashtable works for lookups but not for in-order iteration. Sorting a hashtable takes O(n log n) at flush time. A better choice is an ordered data structure. The most common in practice is a skip list; for example, LevelDB and RocksDB both use one as their default. A radix trie is another elegant option: it keeps keys in lexicographic order naturally, so iterating in order is just a depth-first traversal, and flushing becomes a simple stream with no sorting step needed. A balanced BST works too.

Production implementations typically attach a monotonic sequence number to each entry, so the engine can always determine which version of a key is the most recent, regardless of arrival order.

The memtable doesn’t grow forever. At some point, it gets flushed to disk, and a new empty memtable takes its place. What triggers that flush depends on the implementation: it can be a size limit (a number of entries or a memory threshold), elapsed time, or memory pressure, for example. That flush produces a sorted file on disk called an SSTable, which we’ll look at after the WAL.

The Write-Ahead Log

There is a problem with keeping writes in memory: if the process crashes, everything in the memtable is gone. Any write the client received an acknowledgment for is now lost. That breaks a core database guarantee: durability.

The solution is a Write-Ahead Log, or WAL.

Before writing to the memtable, the engine appends the operation to the WAL, an append-only file on disk. Only after the WAL entry is safely persisted does the engine update the memtable and acknowledge the client. This ordering is what the “write-ahead” in the name refers to: the log is always written before the in-memory state changes.

The WAL is not the final home for data; it’s a safety net. If the engine crashes and restarts, it replays the WAL from the beginning to reconstruct the memtable, recovering any writes that hadn’t been flushed to disk yet.

One subtlety: writing to a file is not the same as persisting it. Operating systems buffer writes in memory before flushing to disk. To guarantee durability, the engine must call fsync() after each WAL entry, forcing the OS to flush its buffers to physical storage. This is not free, though. fsync() adds latency to every write. Production systems often use fdatasync() instead, which persists the data without flushing unnecessary file metadata, keeping WAL appends faster. Many also use a technique called group commit to amortize this cost further: instead of syncing after every write, they batch multiple WAL entries and call fsync() once for the group.

The WAL introduces write amplification: the ratio of data written to disk versus data actually requested by a client. Every byte we write to the database gets written to disk twice: once to the WAL immediately, and once to an SSTable when the memtable is eventually flushed. That cost buys us durability.

SSTables

As we said, when the memtable fills up, it gets written to disk as a Sorted String Table, or SSTable.

An SSTable is an immutable, sorted file. Immutable means it is never modified after creation. Sorted means keys are stored in lexicographic order. Both properties matter:

• Immutability makes SSTables safe to read concurrently without locking.

• Sorted order makes lookups inside a file efficient.

In a simple implementation, an SSTable is just a JSON array of key-value pairs, sorted by key:

[

  { “key”: “apple”, “value”: “1” },

  { “key”: “banana”, “value”: “2” },

  { “key”: “cherry”, “value”: “3” }

]

Production systems use a binary block-based format instead. The SSTable is divided into fixed-size blocks, typically 4 KB, though the exact size varies by implementation. Data blocks hold the actual key-value entries. The SSTable also contains an index block storing the first key of each data block, which makes it possible to binary search for the right block without reading the entire file. In most implementations, the index block is written at the end of the file, since block boundaries are only known after all data blocks have been streamed out. To look up a key, we read the index block, binary search it to find the right data block, fetch that single block from disk, verify its integrity with a checksum, and then binary search within the block. When the index block is not cached, this means most lookups read two disk pages: the index block and one data block. In practice, index blocks are typically kept in memory, so most lookups require only one disk read.

Each data block also carries a checksum computed over the block’s bytes. Before using the data, the engine verifies the checksum. If they don’t match, the block is corrupted, and the read fails safely rather than returning garbage.

As SSTables accumulate, the engine maintains a catalog file (often called a MANIFEST in systems like RocksDB), which is an append-only log listing all existing SSTables in order of creation. This catalog is the engine’s source of truth for what files exist on disk. On startup, the engine reads it to know which files are live, and replays the WAL to restore the memtable. After a successful flush, the old WAL can be discarded. The data is now safely in an SSTable.

Production systems also compress data blocks, typically with a fast algorithm like Snappy, LZ4, or zstd. Compression reduces disk footprint and I/O at the cost of CPU, and it interacts with block sizing: a compressed block may be smaller than a disk page, so implementations often track both logical and physical block sizes.

Reads and Read Amplification

LSM trees are optimized for writes. Reads are where the trade-off shows.

To look up a key, the engine searches in order of recency: first the memtable, then SSTables from newest to oldest. The first match wins. This ordering matters because the same key can appear multiple times across different SSTables. Each write to a key produces a new entry rather than updating the existing one. The newest version is the correct one.

The problem becomes clear as SSTables accumulate. A key that was written once and never updated might still require the engine to search through dozens of SSTables before finding it, or confirming it doesn’t exist. Each SSTable search is a disk read. This is called read amplification: a single logical read triggers multiple physical reads.

For a key that doesn’t exist at all, the engine must check every SSTable before returning a not-found error. That’s the worst case for read amplification, and it gets worse the more SSTables there are.

This is a fundamental tension in LSM trees, and it reflects a deeper principle known as the RUM conjecture: a storage engine can excel at two of reads, updates, and memory efficiency, but not all three at once. LSM trees make a deliberate choice: optimize for updates, accept read amplification as the cost.

The sorted structure also enables efficient range scans. To retrieve all keys between start and end, the engine scans the memtable in order, then merges sorted streams from the relevant SSTables.

Compaction

The answer to accumulating SSTables is compaction.

Compaction is a background process that takes multiple SSTables, merges them into fewer, cleaner ones, and discards the originals. The result is fewer files to search through, which directly reduces read amplification. It also reclaims disk space consumed by redundant entries: if the same key appears in three different SSTables, compaction keeps only the newest version and discards the rest.

One common algorithm is a k-way merge. The engine opens iterators over all SSTables being compacted, each positioned at the first entry. It uses a min-heap to always pull the smallest key across all iterators. When the same key appears in multiple SSTables, the engine picks the version from the newest SSTable and discards the older ones. The merged output is streamed into new SSTable files. In practice, real systems limit the number of SSTables that can participate in a single compaction run to keep resource consumption under control.

Updating the catalog after compaction requires care. The engine must not delete the old SSTables before the new ones are safely written to disk. The safe sequence is: write new SSTables, fsync, write a new catalog pointing to the new files, fsync, then delete the old SSTables. A crash at any point leaves the engine in a recoverable state: either the old files are still referenced by the old catalog, or the new files are referenced by the new catalog.

Compaction is not free. It consumes I/O and CPU in the background, competing with foreground reads and writes. Every byte of data gets rewritten multiple times across its lifetime, adding to write amplification. Tuning when compaction triggers (and how aggressively it runs) is one of the main knobs in LSM tree performance.

Deletions and Tombstones

We might expect deletion to be straightforward: find the key, remove it. In an LSM tree, it is anything but straightforward.

SSTables are immutable. We cannot reach into an existing SSTable and remove an entry. So when a key is deleted, the engine writes a special marker to the memtable called a tombstone, an entry that says “this key is deleted”. It eventually gets flushed to an SSTable like any other write.

During reads, the engine respects tombstones. If a tombstone for a key is found before a value for that key, scanning newest to oldest, the key is treated as deleted, and a not-found error is returned. The tombstone shadows any older value.

The tricky part is knowing when it is safe to discard a tombstone during compaction. Consider this situation: a tombstone for key foo exists in a newer SSTable, and an old value for foo exists in an older SSTable that hasn’t been compacted yet. If we drop the tombstone during compaction without also removing the old value, the old value becomes visible again. Deleted data reappears. This is called data resurrection, and it is a correctness bug.

NOTE: Correctness here means the engine returns what was actually written, not a stale or deleted value. This is different from consistency in the distributed systems sense, which describes the guarantees clients have about which version of data they see across replicas.

The rule is strict: a tombstone can only be dropped when the engine can guarantee that no older value for that key exists anywhere below it on disk. In practice, this means the compaction must include the oldest SSTables that could still hold a shadowed value.

This is one of those details that seems minor until we get it wrong. A storage engine that resurrects deleted data is not a storage engine we can trust. Getting this right requires knowing exactly where older values can hide, which brings us to how SSTables are organized on disk.

Leveling

Basic compaction, merging all SSTables into one flat pool, works but doesn’t scale. As the dataset grows, a flat pool of SSTables means reads still have to check many files. Leveling is the structural answer.

In a leveled LSM tree, SSTables are organized into levels: L0, L1, L2, and so on. Each level has different rules:

• L0 is the landing zone. When the memtable flushes, the resulting SSTable lands in L0. L0 files can have overlapping key ranges: two L0 files might both contain entries for key foo. This is acceptable because L0 files are small and short-lived.

• L1 and deeper levels are different. Each level maintains non-overlapping key ranges across all its files. A given key can exist in at most one file per level. This is the critical property that makes reads efficient: to look up a key in L1, we don’t scan all L1 files. We use the key ranges to jump directly to the one file that could contain it.

When L0 accumulates enough files, a compaction runs to merge L0 into L1. This merge enforces the non-overlapping invariant: L0 files (which may overlap) get merged with the relevant L1 files (which define the ranges), producing new L1 files with clean, non-overlapping ranges. Similarly, when L1 grows too large, a compaction merges part of L1 into L2.

Each deeper level is typically larger by a fixed ratio, for example, 10x. L1 might hold 10 MB, L2 100 MB, L3 1 GB, and so on. Most data ends up in the deepest level. Most compaction work happens between levels.

The benefit is controlled read amplification. To look up a key, we check the memtable, scan all L0 files, then do one binary search per deeper level. The number of deeper levels grows logarithmically with data size. For a dataset with a few levels, that’s a small, bounded number of disk reads, regardless of how many total SSTables exist.

When compaction falls behind and L0 accumulates too many files, the engine may trigger a write stall: new writes are paused until compaction catches up and L0 is drained. This is one of the more painful operational issues in LSM-based systems.

Leveled compaction is also not the only strategy. Tiered compaction, used by Cassandra, for example, takes a different approach: instead of enforcing non-overlapping ranges per level, it groups SSTables of similar size and merges them when a tier grows too large. Tiered compaction generates less write amplification but more read amplification. The right choice depends on the workload.

Bloom Filters

Leveling helps with reads, but there is still one painful case: looking up a key that doesn’t exist.

For a missing key, the engine checks the memtable (not there), checks each L0 file (not there), then checks one file per deeper level (not there). Each check is a disk read. Even with leveling, this adds up.

Bloom filters solve this. A Bloom filter is a probabilistic data structure that can answer one question: Is this key definitely not in this SSTable? It has no false negatives: if the key is in the SSTable, the filter will say so. It can have false positives (occasionally it says a key might be present when it isn’t), but in practice, the false positive rate is tunable and kept very low.

Many implementations attach a Bloom filter to each SSTable, built at creation time from all the keys it contains. The filters are small, a few kilobytes per SSTable, so they can be loaded into memory at startup and kept there.

How does it work? A Bloom filter is a bitset. When a key is added, several hash functions are applied to it, each producing an index into the bitset. The bit at each index is set to 1. To check if a key is in the filter, the same hash functions are applied. If any of the resulting bits is 0, the key is definitely not in the SSTable. No disk read needed. If all bits are 1, the key might be there, and the engine proceeds to read the SSTable.

The practical impact is significant. For a key that doesn’t exist (the worst case), the engine skips almost every SSTable without a single disk read. Only the rare false positive triggers an unnecessary disk read. Read amplification for missing keys drops dramatically.

Some engines take this further and attach Bloom filters not just per SSTable but per data block within an SSTable, enabling even more precise filtering before fetching a block from disk.

Concurrency

Everything described so far assumes a single thread. In reality, a storage engine needs to handle concurrent reads and writes, while flush and compaction run in the background. This is where things get subtle.

The core problem: a flush operation replaces the current memtable with a new one and registers a new SSTable in the catalog. A compaction operation removes old SSTables and registers new ones. If a read is in the middle of searching an SSTable that gets deleted by a concurrent compaction, that’s a crash.

One common solution is a versioned catalog.

A catalog is a snapshot of the engine’s state at a point in time: a reference to the current memtable, the current WAL path, and the current catalog file. Every incoming request acquires the latest catalog version, pins it by incrementing a reference count, performs its work, then releases it by decrementing the reference count.

Background workers (the flush worker and the compaction worker) never modify an existing catalog. Instead, when a flush or compaction completes, they create a new catalog version pointing to the updated memtable and SSTable set. From that moment, new requests acquire the new catalog. Old requests that pinned the previous catalog continue reading from it safely.

An old catalog version is only cleaned up (its SSTables deleted, its WAL file discarded) when its reference count drops to zero. No reader is using it anymore, so it is safe to remove.

This approach keeps foreground reads and writes lock-free in the hot path. Background operations never block requests, and requests never block background operations. They operate on independent catalog versions and only synchronize at the moment of catalog swap, which in many implementations is a single atomic pointer update.

The versioned catalog is also what makes crash recovery clean. On startup, the engine reads the latest catalog file on disk, which always reflects a consistent state: either from before the last flush/compaction, or after. Any SSTables on disk not referenced by the catalog are orphans from an incomplete operation and can be safely deleted.

Summary

• LSM trees optimize for write throughput by turning random disk writes into sequential ones, at the cost of more complex reads.

• The memtable absorbs writes in memory; an ordered structure like a skip list, balanced BST, or radix trie keeps keys sorted for efficient flushing.

• The WAL provides durability: every write is logged to disk before the memtable is updated, enabling crash recovery.

• SSTables are immutable, sorted files produced by flushing the memtable; a binary block format with checksums makes point lookups efficient and reads safe.

• A catalog file tracks which SSTables are live and is updated atomically to ensure the engine always has a consistent view of disk state.

• Read amplification is the fundamental trade-off: finding a key may require searching multiple SSTables, one per level, plus all L0 files.

• Compaction merges SSTables, eliminates redundant entries, and reclaims space, at the cost of write amplification and background I/O.

• Tombstones handle deletions in an immutable structure; they can only be discarded when no older value they shadow still exists on disk.

• Leveling organizes SSTables into levels with non-overlapping key ranges, bounding read amplification to one file lookup per level. Tiered compaction is an alternative strategy that trades less write amplification for more read amplification.

• Bloom filters allow the engine to skip SSTable reads for missing keys with near certainty, eliminating the worst-case read scenario.

• A versioned catalog is one common approach to enabling lock-free concurrent reads and background operations by letting each request pin a consistent snapshot of engine state.

Resources

More From the Distributed Systems Category

• CRDTs Explained

• Availability Models Explained

• The PACELC Theorem Explained

Sources

• The Log-Structured Merge-Tree (LSM-Tree) // The original LSM tree whitepaper.

• Log Structured Merge Tree - ScyllaDB // LSM tree definition from ScyllaDB technical glossary.

Explore Further

• Build Your Own Key-Value Storage Engine

• IO devices and latency

At Google, I’m an SRE on the Google Distributed Cloud team, where the infrastructure stack spans Kubernetes, Borg, distributed storage, virtualization, networking, and more. Over the past months, I’ve been experimenting with ways AI can help not only by automating work away, but also by reducing the cognitive overhead that makes production work quite overwhelming sometimes.

Here are three directions that changed how I thought about the problem.

Situation Awareness

In my team, we have hundreds of dashboards. Kubernetes clusters, Borg jobs, storage metrics, VM utilization, network metrics, etc. Each one tells part of the story.

When something went wrong, and I wanted to understand the current state of the system, I needed to spend a significant amount of time opening tabs and cross-referencing panels to get a complete picture.

This is a fundamentally human bottleneck. Each dashboard was designed to answer a specific question. The question “What is the current situation?” doesn’t map to any single dashboard, and navigating all of them to reconstruct an answer takes time we often don’t have.

Interestingly, this is where AI can change the equation. Instead of navigating dashboards, imagine describing your system to an AI agent with access to your observability stack and simply asking: “What’s going on?” The agent queries across your telemetry data, picks out what stands out, and hands you back a coherent narrative, something you can actually act on. Like: “This specific cluster has an issue with all the containers using distributed storage running on that specific node since 2h.”

This shifts the focus from navigator (opening dashboards one by one) to interpreter (acting on a synthesized summary). And that shift matters: every minute you spend navigating is a minute you're not spending on the actual problem.

Telemetry Archaeology

A few months ago, I was investigating a storage incident on a cluster. The failure itself was clear: a disk issue that surfaced as elevated latency and eventually a service degradation. What wasn’t clear was why it happened when it did.

I used Gemini CLI to navigate the metrics data around the event window. What it surfaced surprised me: the root cause signals had been present in the telemetry hours before the incident triggered any alert. Subtle correlations across metrics that individually looked like noise: disk read latency creeping slightly upward, I/O wait ticking up on specific nodes, a minor memory pressure pattern. Together, they pointed directly at the failure that was coming.

A human reviewing those dashboards in real time would almost certainly have missed it. Each individual signal was within an acceptable range. The pattern only became visible when we looked at all of them together, across time.

This is what I’d call telemetry archaeology: using AI to go back through your metrics data and surface the correlations an alerting system wasn’t designed to catch.

It’s worth being precise about what makes this different from anomaly detection. Anomaly detection tells you when something looks wrong. Telemetry archaeology is about finding the patterns that appear before anything looks wrong at all, relationships that no one thought to encode into an alert, because no one knew they existed until the incident happened.

The practical implication is significant. If these correlations exist in your past incidents, they likely exist in future ones. An AI agent that continuously monitors for these multi-signal patterns could surface a warning (”This looks like the early stages of what happened last time”) long before your system starts showing symptoms.

Incident Co-Pilot

Active incidents can be cognitively brutal. You can be debugging a live system, managing communication with stakeholders, coordinating with other engineers, and trying to remember what you checked 20 minutes ago, all at the same time.

A common consequence is that the engineer with the deepest system knowledge gets pulled out of deep focus to write status updates, summarize what’s been tried, and maintain a running timeline. This work is necessary, but it’s expensive. Every context switch makes it harder to hold the full mental model of the incident in your head. And once that model fragments, rebuilding it takes time you don’t have.

NOTE: This is actually one of the reasons Google developed the IMAG process, with clear role separation: The Incident Commander (IC) coordinates the overall response, the Communications Lead (CL) handles stakeholder updates, and the Operations Lead (OL) focuses on mitigating the issue. The explicit goal is to prevent any single person from being pulled in too many directions at once.

AI can absorb most of this overhead. Think of it as a second brain that’s been in the room the whole time: it tracks what hypotheses have been tested, which ones were ruled out and why, what changed in the system during the incident window, and what hasn’t been explored yet. When a new engineer joins the investigation, instead of spending ten minutes getting them up to speed, you ask the AI for a summary.

AI’s role here is handling the administrative layer of the incident: the parts that pull you out of flow, so you can stay in the problem instead of constantly being yanked out of it.

I’ve been using AI this way during my own shifts. Even without a purpose-built tool, maintaining a running log with AI (e.g., what we’ve tried, what we know, what’s next) noticeably changes how an incident feels.

Summary

• The common “AI for production” narrative focuses on automation and replacement; cognitive augmentation is the underexplored angle.

• Situation awareness: AI can synthesize across hundreds of dashboards to answer “What’s the current situation?” in seconds, shifting your role from navigator to interpreter.

• Telemetry archaeology: AI can surface hidden correlations across metrics that individually look like noise, revealing root cause signals that were present hours before any alert fired.

• Incident co-pilot: AI can absorb the administrative layer of an active incident (status updates, running timeline, hypothesis tracking), keeping the engineer in deep focus instead of constant context switching.

• None of this requires replacing the engineer. The value is in extending what one person can hold in their head under pressure.

Resources

More From the Reliability Category

• Reliability

• Resilient, Fault-tolerant, Robust, or Reliable?

• Lurking Variables

Explore Further

• Gemini CLI

• Google Site Reliability Engineering: Incident Management Guide

• The future of software engineering is SRE

A Cache for Latency

Latency is the time between when a request is sent and when a response is received. A cache for latency exists to reduce the average latency of a service.

The classic access pattern looks like this1:

1. We check the cache first.

2. On a cache hit, we return the data directly without touching the backend.

3. On a miss, we go to the backend, return the result, and store it in the cache for future requests.

Why does this reduce latency? The cache keeps data in memory, which is significantly faster to read from than a remote database that may involve network round-trips, disk I/O, and query execution. On a hit, all of that work is skipped.

In Soft vs. Hard Dependency, we introduced two kinds of dependencies:

• A soft dependency is a non-critical dependency for the service to operate properly.

• A hard dependency is a critical dependency for the service to operate properly.

A cache for latency is a soft dependency. If the cache becomes unavailable, requests fall through to the backend. The system keeps working, just at a higher latency. Keep this in mind, because it’s the key difference we’ll come back to.

A Cache for Capacity

A cache for capacity exists to serve higher throughput than the backend can handle on its own.

The access pattern is identical to the latency case: cache first, then backend on a miss.

So what actually makes these two different?

The difference is not in the code; it’s in what the backend can absorb. In a capacity scenario, the backend would be overwhelmed if it received all the traffic directly. The cache absorbs a large portion of the requests, keeping the backend load manageable.

This changes the nature of the dependency. If the cache goes down, the backend is suddenly hit with all the traffic it was previously shielded from. Whether the system survives depends on the backend’s own capacity. If the backend can scale fast enough, the cache is still a soft dependency: there will be a rough period, but the system recovers. If the backend can’t cope with the load, the cache becomes a hard dependency. Without it, the system fails.

When a Latency Cache Becomes a Capacity Cache

Here’s a question worth asking: if the access pattern for both types is identical, how do we know which one we have?

In most cases, caches are introduced to reduce latency. But here’s what can happen over time:

• Our system is stable.

• Cache hit rates are high, backend load is low.

• Traffic grows. The backend load stays low because the cache is absorbing most of it. Nothing breaks. No alerts fire.

• Six months pass. Nothing has changed, no code, no configuration, no architecture decision. And yet the cache is no longer reducing latency. It’s keeping the backend alive.

The cache didn’t change. The code didn’t change. The system grew around the cache, and the cache quietly became load-bearing.

The same risk appears when a cache goes cold. For example:

• A migration to a new cache instance

• A data format change that requires purging existing entries

• A cache restart after maintenance

Any of these can produce a large wave of cache misses in a short window. If we were running a latency cache, we would see higher latency for a while. If we were running a capacity cache, we would see a traffic spike that the backend can’t absorb.

The unsettling part is that the code is identical in both cases. The difference only becomes visible at failure time.

How to Manage This Risk

The root problem is that teams often don’t know which type of cache they’re running. They built it for latency, and that’s still how they think about it, even as the system outgrows that assumption.

A few approaches help here:

• Periodically ask: could the backend handle the current traffic if the cache were completely removed? Load testing without the cache, or estimating backend capacity against current traffic levels, gives you a concrete answer.

• Treat cache hit rate as a meaningful operational signal, not just a performance metric. A sustained drop in hit rate means the backend is absorbing more traffic than usual. If that trend continues, it’s an early warning that you may be drifting toward a capacity problem.

• When migrating a cache or invalidating a large portion of its data, warm the new cache before routing live traffic to it. This prevents a cold-start burst from hitting the backend all at once.

• Finally, once we recognize that a cache is operating as a capacity cache, we should treat it accordingly. It’s no longer optional infrastructure and it deserves proper alerting and a clear plan for what happens if it goes down.

Summary

• A cache for latency serves data from memory to reduce average response time. It is a soft dependency: if unavailable, the system degrades in latency but continues to work.

• A cache for capacity absorbs traffic that the backend couldn’t handle on its own. It can be a soft or a hard dependency, depending on whether the backend can absorb the load without it.

• Both types share the same access pattern, which makes them easy to confuse.

• A latency cache can silently become a capacity cache as traffic grows, without any code change.

• When a capacity cache goes cold or fails, the backend can be overwhelmed. Hit rate monitoring, periodic load testing, and cache warming are practical ways to manage this risk.

Resources

More From the Distributed Systems Category

• Availability Models

• Safety and Liveness

• The PACELC Theorem

Sources

• The Three Types of Cache

Explore Further

• Cache stampede

The Problem

Salvatore Dipietro ran pgbench (PostgreSQL’s standard benchmarking tool) on a Graviton4 processor with 96 vCPUs. The workload was a benchmark doing simple updates at scale factor 8,470 (i.e., roughly a 847 million row table), simulating 1,024 clients and 96 threads. A serious, high-parallelism load designed to stress the system.

The results were striking. Linux 7.0 delivered roughly half the throughput of Linux 6.x on the same hardware and workload:

• Linux 6.x: 98,565 transactions per second

• Linux 7.0: 50,751 transactions per second

To find where the time was going, Dipietro ran perf, a Linux profiling tool that samples what the CPU is actually doing. The result was unambiguous:

|- 56.03% - StartReadBuffer
   |- 55.93% - GetVictimBuffer
      |- 55.93% - StrategyGetBuffer
         |- 55.60% - s_lock         # 55% of CPU

55% of the machine’s CPU time was spent inside a single function: s_lock. The culprit was traced back to a change in how Linux 7.0 schedules processes. Let’s start there.

What Is Preemption?

When multiple threads run on a machine, the OS needs to share the CPU between them. That’s the scheduler’s job. But the scheduler also decides something subtler: when to interrupt a running thread and hand the CPU to another. That decision is called preemption, and the answer varies depending on how the kernel is configured.

Before Linux 7.0, there were three options:

• PREEMPT_NONE: The kernel almost never interrupts a running thread. A thread runs until it voluntarily gives up the CPU: when it makes a syscall, blocks on I/O, or explicitly sleeps. This was the traditional server default with fewer context switches, higher throughput, and predictable behavior under load.

• PREEMPT_FULL: The kernel can interrupt a running thread at almost any safe point, even if it is in the middle of doing useful work. This means a thread never has to wait for the current one to finish its slice before getting CPU time, which reduces response time but increases context-switch overhead. Historically, the desktop default, where responsiveness matters more than raw throughput.

• PREEMPT_LAZY: Introduced in Linux 6.12 as a compromise between the two. The scheduler can interrupt threads, but tries to wait for natural boundaries rather than cutting in aggressively. The intent is to approximate PREEMPT_NONE‘s throughput behavior while still allowing preemption when needed.

In Linux 7.0, PREEMPT_NONE was removed as an option on modern CPU architectures, leaving only PREEMPT_FULL and PREEMPT_LAZY. Indeed, PREEMPT_LAZY was designed to be a drop-in replacement on throughput workloads, and for the vast majority of server software, it is. But PostgreSQL hit a specific case where the difference is catastrophic, and to understand why, we need to look at how PostgreSQL manages memory.

PostgreSQL Memory Management

PostgreSQL, like most databases, doesn’t store data as rows in a flat file. Instead, it uses a fixed-size abstraction called a data page (8 KB by default) as its basic unit of storage. Everything on disk (e.g., table rows, B-tree index nodes, metadata) is stored in these pages. A table with millions of rows is ultimately a large sequence of data pages on disk.

Reading from disk is slow. So PostgreSQL maintains a shared buffer pool, a large region of shared memory that caches recently read data pages. The more of the working set that fits in the buffer pool, the less disk I/O is needed.

When a client connects to PostgreSQL, the server spawns a dedicated process to handle that connection, called a backend. Every backend that needs a data page not already in the buffer pool has to first read it from disk, then find a buffer to store it in: either one that is already free, or one currently holding another page that can be evicted. The job of finding that buffer falls to a single crucial function called StrategyGetBuffer.

To coordinate access to the buffer pool across hundreds of concurrent backends, StrategyGetBuffer uses a spinlock.

Spinlocks in PostgreSQL

A spinlock is a locking mechanism built on a simple idea: instead of going to sleep while waiting for a lock to become available, a process just keeps checking in a tight loop (it spins):

while (!try_acquire_lock()) {
    /* Keep checking, burn CPU */
}

Why would we ever want that? For very short critical sections, the overhead of putting a thread to sleep and waking it back up can be more expensive than just “spinning“, meaning actively waiting. If we know the lock holder will be done in nanoseconds, spinning is faster than sleeping.

The key assumption behind spinlocks is the following: the thread holding the lock will release it very soon. Nobody is going to preempt that thread in the middle of a 20-nanosecond critical section. The holder will finish and release the lock before anyone has time to notice.

StrategyGetBuffer uses a single global spinlock to protect the critical section where it selects a buffer. On a 96-vCPU machine with 1,024 clients all hammering the database, every backend competes for the same lock, and any time it takes longer than expected to release, all of them burn CPU spinning.

But why did the Linux 7.0 preemption change make it so much worse? The answer lies in how memory works at the hardware level.

Virtual Memory and the TLB

Every process in Linux, including PostgreSQL, works with virtual memory addresses. For example, the address 0x7fff1234 in one process is a completely different memory from the same address in another process. The hardware translates virtual addresses to physical addresses using a data structure called the page table, maintained by the kernel in memory.

A page table is a multi-level tree, so a single address translation requires several sequential memory reads to walk it. Doing that for every memory access would be impossibly slow.

Instead, CPUs have a small hardware cache for recent translations called the Translation Lookaside Buffer (TLB):

• When a process accesses an address it has accessed recently, the TLB already has the translation, and the memory access proceeds quickly.

• When a process accesses an address it hasn’t seen before, it gets a TLB miss: the CPU has to walk the page table, find the physical address, and store the translation in the TLB. That takes time.

There is one more concept to introduce. When PostgreSQL starts, it allocates the shared buffer pool as a large virtual memory region. But allocating virtual memory and having physical memory ready to use are two different things. Indeed, Linux uses a principle called lazy allocation: the allocation is noted, but the actual physical pages are only mapped on first access.

The first time any code touches a previously-unmapped virtual address, a minor page fault occurs: the kernel allocates a physical page and stores the mapping. That takes microseconds, orders of magnitude slower than a regular read or write where the page is already mapped.

The Problem with 4 KB Pages

When a process accesses memory for the first time, the kernel doesn’t map it byte by byte. Instead, it maps memory in fixed-size chunks called memory pages via the page table.

NOTE: We already used the word “page” to characterize data pages, meaning how PostgreSQL organizes data on disk into fixed-size 8 KB blocks. This is a different concept than a Linux page, which is the unit the kernel uses to manage physical memory.

By default, a Linux memory page is 4 KB. PostgreSQL's shared buffer pool, like all memory on Linux, is backed by Linux memory pages under the hood. In Dipietro’s benchmark, the shared buffer pool was configured to 120 GB via the shared_buffers parameter, which at 4 KB per Linux memory page means roughly 31 million memory pages. Therefore, 31 million potential first-touch page faults.

Now let’s consider what happens inside StrategyGetBuffer. Each backend acquires the spinlock to find a free slot in the buffer pool. To do so, it reads or writes shared memory. If that region of shared memory hasn’t been touched yet, accessing it triggers a minor page fault, meaning that the kernel has to allocate a physical memory page and store the mapping.

During a long benchmark with a 120 GB shared buffer pool, new regions keep entering the working set throughout the run, so these faults happen constantly, not just at startup. And when a fault occurs while a backend is holding the spinlock, the consequences are severe.

Indeed, we discussed that the key assumption behind spinlocks is that the lock will be released very soon. In that case, the assumption breaks: the holder is stuck inside the kernel fault handler while it stores a physical memory page mapping, and every other backend on the machine is spinning, burning CPU, waiting for a lock that won't be released until the faulting process resumes.

The impact of a fault when the lock was acquired depends on the preemption model. Let’s consider the following example. Backend A acquires the lock but triggers a page fault. Meanwhile, backends B, C, and D arrive and try to acquire the lock. Since they can’t, they spin, burning CPU on a tight loop while waiting for backend A to release the lock.

• With PREEMPT_NONE (before Linux 7): Once backend A enters the fault handler, the kernel handles the fault. Since PREEMPT_NONE avoids voluntary rescheduling points, backend A is unlikely to be scheduled away before the fault resolves and the lock is released. The spinners wait a bit longer than expected, but the damage is limited.

• With PREEMPT_LAZY (Linux 7 and beyond): The scheduler may decide to preempt backend A while it’s still inside the fault handler, scheduling another process in its place. Backend A won’t resume until the scheduler hands control back to it, which can take some time, even after the fault is fully handled:

The spinlock hold time goes from “duration of the fault” to “duration of the fault + time waiting for the scheduler.” And that extra wait, let’s call it t, is not just t of wasted CPU; instead, it is t multiplied by every backend currently spinning. In the previous example, backends B, C, and D each burn t extra cycles, making the total waste 3t. On a 96-vCPU machine with hundreds of backends, that multiplier is devastating. That's how the benchmark ended up with 56% of the CPU burning in s_lock.

That extra time waiting for the scheduler was the root cause of the issue.

Huge Pages to the Rescue

Fortunately, there is an option to overcome this issue in PostgreSQL.

The main variable we discussed was shared_buffers, 120 GB in the benchmark, meaning roughly 31 million memory pages. But there is another variable we can adjust: the size of a memory page.

As we said, it defaults to 4 KB, but the kernel supports larger pages called huge pages. On x86_64 and ARM64, the supported sizes are 2 MB and 1 GB:

• 4 KB pages: ~31,000,000 potential page faults

• 2 MB huge pages: ~61,440 potential page faults

• 1 GB huge pages: ~120 potential page faults

Increasing the size of a memory page reduces the number of potential page faults but also reduces TLB pressure. Indeed, far fewer entries need to cover the same memory, so the working set fits comfortably in the TLB, meaning far fewer TLB misses and page table walks on the hot path. Overall, StrategyGetBuffer stops triggering faults while holding the lock. The lock holder finishes quickly. The other backends wait microseconds instead of milliseconds. The regression disappears.

NOTE: Setting huge pages in PostgreSQL is controlled by the huge_pages configuration parameter, which accepts three values: off, on, and try (the default). With try, PostgreSQL uses huge pages if available and silently falls back to 4 KB pages otherwise. Use on instead so PostgreSQL fails to start rather than running misconfigured without you noticing. The size of the huge pages themselves is a Linux configuration.

However, setting huge pages is not without tradeoffs. Huge pages are pre-allocated and reserved upfront, meaning that memory is no longer available to the rest of the system even if PostgreSQL isn’t using it all. There is also a memory waste concern: a huge page is allocated as a whole, so if only a fraction of it is used, the rest is wasted. For most production PostgreSQL deployments with large shared_buffers, these tradeoffs are probably worth it, but they are good to know about.

What Happens Next?

Peter Zijlstra, the Intel kernel engineer who authored the preemption change, proposed a fix: PostgreSQL should adopt Restartable Sequences (rseq), a Linux kernel facility that lets userspace code detect whether it was preempted or migrated during a critical section and restart it if so. PostgreSQL's spinlock paths would use rseq to detect preemption and retry, avoiding the scenario where a preempted lock holder stalls all waiting backends.

The PostgreSQL community’s response was not enthusiastic. Using a kernel facility specifically to recover performance that PostgreSQL had for free before Linux 7.0 is a tough sell. Meanwhile, it would also go against kernel’s long-standing principle of not breaking userspace: if software worked correctly before a kernel upgrade, it should work correctly after.

Summary

• Linux 7.0 removed PREEMPT_NONE on modern CPU architectures, leaving only PREEMPT_FULL and PREEMPT_LAZY. On most distributions, the default shifted to PREEMPT_LAZY.

• An AWS engineer benchmarked PostgreSQL on a 96-vCPU Graviton4 and found throughput cut in half on Linux 7.0, with 55% of CPU burning inside a single spinlock in StrategyGetBuffer.

• The root cause is minor page faults occurring while a backend holds the spinlock. With 4 KB memory pages backing a 120 GB shared_buffers, there are up to 31 million potential first-touch faults throughout a benchmark run.

• Under PREEMPT_NONE, the faulting process resumed quickly and released the lock. Under PREEMPT_LAZY, the scheduler may preempt it mid-fault, extending the hold time and causing every waiting backend to keep spinning.

• Enabling huge pages (2 MB or 1 GB) reduces the number of potential faults by orders of magnitude and eliminates TLB pressure, making the regression disappear.

Resources

More From the Systems Category

• Linux Soft vs. Hard Lockup

• Instruction Pipelining Explained

• Simultaneous Multithreading Explained

Sources

• [PATCH 0/1] sched: Restore PREEMPT_NONE as default

• AWS Engineer Reports PostgreSQL Performance Halved By Linux 7.0, But A Fix May Not Be Easy

• PREEMPT_NONE Is Dead; Your Postgres Probably Doesn’t Care

Explore Further

• The long road to lazy preemption

• Buffer Manager

• Huge pages

• Restartable Sequences

Introduction

Have you ever fixed an incident, only to see it come back two weeks later? Or made a change that improved one metric while quietly degrading another? Or spent months firefighting without ever feeling like things were actually getting better?

These aren’t signs of bad engineering. They’re signs of reacting to events without understanding the structures that produce them.

Understanding those structures requires a different kind of thinking, and that’s exactly what systems thinking is: the ability to shift from reacting to events through responsive patterns of behaviors to generating improved systemic structures.

This post is an introduction to systems thinking, covering the core concepts through a real example from my experience at Google.

What Is a System?

First, let’s define what a system is. In essence, a system is:

• A set of elements

• Interconnected

• To achieve something

Distributed systems are an obvious example. For example, a 3-node, single leader database is composed of:

• 3 nodes (elements)

• Connections from the leader to the replicas (interconnections)

• With the goal of storing data reliably over time

Interestingly, this is why distributed systems can surprise even their own designers: add enough nodes, replication lag, and competing writes, and the system starts behaving in ways no single component would predict.

Stocks and Flows

To reason about how systems change over time, we need two important concepts:

• A stock is an accumulation of material or information that has built up in a system over time. For example: the number of machines available in a cluster, the size of a message queue, the amount of technical debt in a codebase.

• A flow is what changes a stock: material or information entering or leaving it. For example: machines being added or removed from service, messages being enqueued and consumed, or requests being received and processed.

The key thing to keep in mind: stocks take time to change because flows take time to flow. You can’t instantly restore machine availability or drain a queue with a single action. This has real consequences for how systems behave under pressure. We will come back to it.

Feedback Loops

One of the most important concepts in systems thinking is the feedback loop.

A feedback loop is what the system does automatically because its own result feeds back into it. Said differently: If A causes B, then B influences A.

Let’s take a concrete example. Suppose you live in a house with a central thermostat set at 20°C. It turns the heating on when the temperature drops to 19°C, and off when it reaches 21°C.

The feedback loop works like this:

• A: Temperature change

• B: Thermostat turns heating on or off

The thermostat turning on or off (B) is caused by the temperature change (A). But the temperature change (A) is in turn influenced by the thermostat (B). Each effect feeds back into its own cause. This is a feedback loop.

Balancing vs. Reinforcing Feedback Loops

There are two kinds of feedback loops.

• A balancing feedback loop resists change: It pushes the system back toward a goal or limit. Think of it as a stabilizer: when something moves away from the target, the loop acts to bring it back. The thermostat is a perfect example. As the temperature drifts away from 20°C, the thermostat reacts, and the system returns to equilibrium.

• A reinforcing feedback loop amplifies change: More leads to more, less leads to less. An action produces a result that drives more of the same action, generating growth or decline at an accelerating rate. The YouTube algorithm is a clear illustration: the more a video is viewed, the more the algorithm surfaces it; the more it’s surfaced, the more views it gets.

More formally, we can have 4 cases of feedback loops:

1. Balancing ceiling: If A↑ causes B↓, then B↓ influences A↑

2. Balancing floor: If A↓ causes B↑, then B↑ influences A↓

3. Reinforcing growth: If A↑ causes B↑, then B↑ influences A↑

4. Reinforcing collapse: If A↓ causes B↓, then B↓ influences A↓

The more feedback loops a system contains, the more complex and surprising its behavior becomes, especially when those loops interact.

Delays

An often overlooked but critical property of feedback loops is the delay between an action and its effects.

Delays are pervasive in systems and strong determinants of behavior. When the gap between action and effect is long, two things happen:

• Foresight becomes essential: Acting only when a problem becomes obvious means missing the window to address it early.

• Oscillations become likely: We overreact because the system hasn’t had time to respond, then overreact again in the other direction.

Think of an autoscaler that takes 3 minutes to provision new instances. By the time the new capacity is ready, the traffic spike has already peaked. The window to act had opened before the problem was even visible on the dashboard. This is why foresight matters: when there is a significant delay between action and effect, reacting to what you see now means always acting too late.

And the consequences compound. The autoscaler, still responding to the old signal, overshoots. Then it sees too much capacity and scales down, right before the next spike arrives. One example, two problems: a system that needed foresight got a reaction, and then oscillated because of it.

The delay didn’t change the goal. It made the system work against itself.

System Boundaries

System boundaries are artificial. They help us frame a problem, but in reality, everything is interconnected. The boundaries we draw determine what we see and, therefore, what we miss.

Consider a microservices architecture in which each team owns a service. Every team has solid SLOs, careful on-call rotations, and clean dashboards. And yet end-to-end latency keeps creeping up, and users are complaining. Each team looks at its own service and sees green. The problem is that the boundary is wrong; no one is looking at the system as a whole.

This is one of the most common traps in engineering: optimizing within a boundary while the real issue lives outside it. Before changing a system, it is worth asking: Am I looking at the right boundary?

The Iceberg Model

When something goes wrong in a system, what do we actually see? Usually just the surface: an incident, a spike, an outage. The iceberg model gives us a way to think beneath it.

The model has four levels:

• Events are what’s visible: the incident alert, the latency spike on the dashboard. This is where most of our attention goes, and where reactive thinking lives.

• Patterns and trends are what you find when you zoom out. Has this happened before? At what frequency? Under what circumstances? Patterns reveal that what felt like a one-off event is actually part of a larger rhythm.

• Structure is the underlying system design: the feedback loops, the incentives, the processes that produce the patterns. You can’t fix a pattern without understanding the structure that generates it.

• Mental models are the beliefs and assumptions that shaped the structure in the first place. They’re the hardest to see and the hardest to change.

Most incident response lives at the event level. Systems thinking asks us to go deeper. As an SRE, this model resonates: we’re trained not just to react to incidents but to understand the why: the patterns, the structures, and eventually the assumptions that caused them.

A Concrete Example: Safe Removal Service

Let me now bring all of these concepts together through a concrete example from my previous role at Google, where I worked on the systems powering Google’s ML infrastructure.

I was heavily focused on a system called the Safe Removal Service1 (SRS). This service had a simple API and one core responsibility: to say yes or no when another system requested permission to disrupt a given entity. Indeed, most disruptive services at Google, the ones that reboot machines, drain jobs, or take clusters offline, were designed to ask this service before acting.

In our context, the key constraint was preserving capacity, meaning ML TPUs and GPUs. For example, within a given cluster, at least 90% of TPUs must remain available at all times. So if 95% were currently available, SRS could approve disruptions, as long as availability didn’t drop below 90%.

NOTE: The threshold values and other details have been altered for confidentiality reasons.

The API was deliberately simple:

• “Can I reboot this machine?” → Yes/No

• “Can I drain this job?” → Yes/No

• “Can I take down this cluster?” → Yes/No

Identifying the Feedback Loops

SRS implemented several balancing feedback loops. For example, when available capacity dropped toward 90%, the service would start refusing disruptive requests, pushing availability back up. This was the primary loop: a governor that kept the system in a safe zone.

There was also an implicit reinforcing loop on the positive side: by allowing maintenance to proceed when capacity was healthy, the service enabled machines to be upgraded, patched, and kept in good shape, which in turn kept capacity high.

So far, so good. But here’s where it gets interesting.

The Hidden Reinforcing Loops

The balancing loop protected current capacity. What it didn’t account for was what happened when capacity was already constrained.

When available capacity hovered near 90%, SRS would block most maintenance requests. Machines couldn’t be patched. Hardware with known error trends couldn’t be swapped. Security upgrades were deferred. Maintenance debt accumulated, silently, invisibly.

This created a first hidden reinforcing loop:

Less capacity → Deferred maintenance → More failures → Even less capacity

The balancing loop was actively feeding the very problem it was trying to prevent.

A second reinforcing loop emerged from human behavior:

Low capacity → More incidents → Bypass mechanisms invoked → Riskier actions taken → Capacity lower still

When the system was under stress, operators would sometimes override SRS to unblock critical work. Each bypass, reasonable in isolation, eroded the safety margins that the balancing loop was designed to protect.

The Goal Was Inaccurately Defined

There’s a principle from Thinking in Systems that describes this precisely:

SRS was measuring the right-looking metric: current capacity. But the current capacity was not the same as the real health. A cluster at 92% availability, accumulating maintenance debt and hardware errors, was far more fragile than a cluster at 91% that was fully patched and stable. The balancing loop couldn’t tell the difference.

The deeper fix wasn’t just tuning the threshold. It was making the controller health-aware, not just capacity-aware. Rather than gating only on “% available right now,” the system needed to incorporate slow indicators: maintenance backlog growth rate, share of fleet on known-bad firmware versions, hardware error trendlines, override and bypass rates.

By the time the reinforcing loops made their effects visible, the stock (cluster health) had already been degrading for weeks. The delay between cause and effect made the problem invisible until it was expensive to fix. This example was not about a flawed design. It was about a structure that, taken as a whole, was quietly working against itself.

Summary

• A system is a set of elements interconnected to achieve a goal.

• Stocks are accumulations that change over time through flows; stocks take time to change.

• A feedback loop occurs when an effect feeds back into its own cause.

• Balancing feedback loops resist change and push toward equilibrium; reinforcing feedback loops amplify change.

• Delays between action and effect can cause oscillations and make problems invisible until too late.

• System boundaries are artificial; the boundary we draw determines what we see and miss.

• The iceberg model: events are visible, but patterns, structure, and mental models lie beneath.

• System goals must reflect real welfare, not just what’s measurable; inaccurate goals lead to unwanted behaviors.

• A well-designed balancing loop can mask hidden reinforcing dynamics. The most dangerous moment is when a system appears to be working.

Resources

More From the Distributed Systems Category

• Working on Complex Systems

• Probabilistic Increment

• Consensus

Sources

• Thinking In Systems

• Learning Systems Thinking

Explore Further

• Leverage Points: Places to Intervene in a System

❤️ If you enjoyed this post, please hit the like button.

💬 Have you ever built or maintained a system that looked healthy on the dashboard while something was quietly accumulating underneath?

Leave a comment

An Introduction to Matter

To start, what is matter? Matter is made up of molecules, and molecules are assemblages of atoms, the building blocks of matter. For example, water is an H₂O molecule: 2 hydrogen atoms and 1 oxygen atom.

An atom is itself composed of a nucleus and electrons, which carry a negative charge and orbit around it.

The nucleus contains two types of particles:

• Protons, which carry a positive charge, naturally repel each other.

• And neutrons, which carry no electric charge and act as a kind of “glue,” helping to keep the nucleus stable.

The attraction between electrons (−) and protons (+) keeps the whole thing in a stable state.

On the other hand, too few or too many neutrons relative to the protons, and the nucleus becomes unstable. It will eventually decay by emitting energy. This is the principle of radioactivity. Carbon-14, for example, is slightly unstable. It decays slowly and predictably. This predictability allows it to be used as a clock to date ancient elements.

One might think that when touching a solid object, like a table, for instance, what gives the table its solidity is that it is “filled” with matter, preventing our finger from passing through. Yet, if we took the nucleus and enlarged it into a marble and placed that marble on a football pitch, the electrons would be orbiting at the level of the stands.

An atom is therefore almost entirely empty. Solid matter is almost nothing, and what gives this impression of solidity are forces between atoms called electromagnetic forces.

The Fundamental Forces in the Universe

The universe is made up of 4 and only 4 fundamental forces:

• Gravity: Attracts everything with mass toward everything else with mass.

• The strong nuclear force: It glues protons and neutrons together inside the nucleus.

• The weak nuclear force: Responsible for certain radioactive decays. It is what allows a neutron to transform into a proton (or vice versa).

• And the electromagnetic force.

If we focus on this last one, it is the one that:

• Attracts opposite charges

• And repels identical charges.

Unlike the two nuclear forces, which only act inside the nucleus, the electromagnetic force has an infinite range. That is why it is the one that governs interactions between atoms at our scale.

It is therefore the electromagnetic force that creates the illusion of solidity. When we touch a table, it is the electrons in our hand and those in the table that repel each other. We never truly touch anything.

Light: Wave and Particle

Let’s now talk about light. So, what is light? It is an electromagnetic wave, a disturbance of the electric and magnetic fields that propagates through space.

Light is a spectrum. Indeed, so-called visible light, the light our eyes can perceive, is only a tiny portion of what exists. The full spectrum is called the electromagnetic spectrum:

Radio wave → Microwave → Infrared → Visible light → UV → X-rays → Gamma rays

When a radio picks up radio waves, it is therefore picking up light, invisible due to its frequency. Indeed, what varies across the electromagnetic spectrum is the frequency of the wave, and therefore its energy.

But light hides a surprise: it is also a particle.

NOTE: A particle can be summarized as follows: an indivisible packet of energy.

We know it is a particle thanks to Einstein in 1905 (for which he received his only Nobel Prize, not for relativity). When a light bulb emits light, it emits specific particles called photons.

When we vary the intensity of that light bulb, one might assume it is the energy of the photon that varies, but that is not the case. The energy of each photon is fixed by its frequency. The higher the frequency of a photon, the more energetic each photon is. That is why, for example, UV rays burn the skin.

What makes a light bulb emit more light is the increase in electric voltage, which therefore produces more photons. It is the quantity of photons that makes a light bulb shine more or less.

In flight, the photon behaves like a wave: it propagates, it oscillates, and it can interfere with other photons. But when it comes into contact with matter, it behaves like a particle: it interacts in one single hit, in one single place.

When a photon collides with matter, it can either be:

• Absorbed: The photon ceases to exist. Its energy is transferred to an atom, which moves to a higher energy level. This is what an eye does: it absorbs the photon and converts it into an electrical signal.

• Reflected: Technically, this is not a true reflection because it is not the same photon that leaves. The atom absorbs the photon and then re-emits a photon of the same energy in a different direction.

NOTE: What determines whether a photon is absorbed or reflected depends on the energy levels of the electrons in the atoms of the surface. If the photon’s frequency matches an available energy level, the atom absorbs it. Otherwise, the photon is re-emitted. That is why glass is transparent, why the retina absorbs light, and why a mirror reflects almost everything.

The Young’s Double-Slit Experiment

We have seen that light is a wave. But how do we know this? This is where Young’s double-slit experiment comes in, and it is this very experiment that will lay the foundations of quantum physics.

Young’s experiment, carried out for the first time in 1801, is as follows:

• A laser projects photons (light)

• A wall with two small slits, A and B

• A screen behind to detect where the photons land

If light were a “packet” of something, we would see the following result:

Yet, the result of Young’s double-slit experiment is as follows:

We obtain what is called an interference pattern. The wave passes through both slits simultaneously, splits into two, and these two waves meet on the other side.

When two waves meet, they add up or cancel out depending on their respective phase:

• Two crests meeting → they add up → bright zone

• A crest meeting a trough → they cancel out → dark zone

The result is an alternating pattern of bright and dark bands on the screen: that is an interference pattern.

What If We Did the Same With Matter?

In the 20th century, researchers then had an idea: apply Young’s experiment no longer by projecting photons (light) but electrons (matter). The experiment is therefore similar, but instead of a laser, an electron gun is used to then measure on the screen where the matter lands.

Obviously, with this experiment, we are going to get two bands of matter, right?

Well, still no! An interference pattern is observed as well. This result was not a complete surprise to everyone. In 1924, physicist Louis de Broglie had already theoretically proposed that matter, like light, could have a wave-like nature. But this time, it's not a wave-like light; it's a probability wave.

This is one of the greatest discoveries in quantum physics: at the atomic level, the particle has no defined position. The position of a particle is determined by a function called the wave function, ψ(x), which describes the probabilities of finding that particle at a given location in time.

A small clarification on this concept of undefined position to make sure the concept is clear, because this is the moment where our rational brain can start to “let go.”

Let’s take a coin for a coin toss. We throw it in the air and hide the result. We are in a state of uncertainty, but this uncertainty is called epistemic. We do not know the result (heads or tails) because we have not looked yet, yet that result already exists.

For a particle in the quantum world, the uncertainty is called ontological. It is not that we lack information about the position of the particle; it is that this position simply does not exist yet. This is what is called quantum superposition: an unmeasured particle exists in multiple states simultaneously.

However, measurement changes everything. When we measure the position of a particle, we will find it in one of the possible positions described by the wave function. We then say that the wave function “collapses” because it restricts the possibilities into a single real state.

As an analogy, it is a bit like Minecraft. A default Minecraft map is 60 million x 60 million blocks. For the initial loading, the server does not generate the entire map. It only generates the world around the observer, i.e., the player. However, when the player moves, they force the server to generate the world's continuation.

Where this analogy reaches its limits is that the generation of the Minecraft world, even if it is random, is still deterministic because each world has its own seed. The quantum world, on the other hand, appears to be purely random, meaning without hidden information.

Let’s return to Young’s experiment. What would happen if, when a particle passes through a slit, we placed a detector there to observe which slit the particle goes through? We recall that a wave passes through both slits at once.

This is the moment where the brain completely lets go: observing the particle changes the result of the experiment. Indeed, observing that particle “forces” it to have a defined position, and it then behaves like a classical marble. The result, therefore, gives us two bands of matter.

To summarize what we have seen so far: an unobserved particle exists as a probability wave, in multiple positions simultaneously. As soon as we measure it, this wave collapses, and the particle ends up at a precise location.

But then, why do we never see this in everyday life? The answer is decoherence.

Decoherence

Quantum superposition is only possible as long as a particle remains isolated from its environment. As soon as it interacts with anything, another atom, a photon, an electric field, that interaction constitutes a measurement in the quantum sense. The wave function collapses, and the particle ends up in a precise state.

An isolated electron in a vacuum can remain in superposition. But a macroscopic object like a table is made up of billions upon billions of atoms that permanently interact with the surrounding air, light photons, and electromagnetic fields. These interactions occur billions of times per second. The superposition collapses instantaneously before we can even observe it.

That is why quantum physics is only observable at the atomic scale. And that is also why a single electron in a transistor behaves very differently from an object we can hold in our hand.

The Key is Information

OK, so the original Young’s experiment with light produces an interference pattern because light is a wave. The variation with electrons (or indeed subsequently other elements such as atoms) also produces an interference pattern, which proves that matter is a wave, but this time a probability wave. When we measure the result, we change the result of the experiment because we force the particle to “choose” its position.

But incidentally, how does this measurement work in the experiment? It works thanks to photons. Indeed, when the electron passes through one of the slits, we project a photon which will interact with the electron and be re-emitted in a direction that allows us to deduce which slit the electron went through.

Researchers wanted to know what would happen if they performed the exact same experiment, measuring which slit the particle went through, but this time, instead of reading the information encoded in the orientation of the photon, they destroyed that information.

And here, another surprise: if we destroy the information, we return to an interference pattern. It was as if, since we were not using that information, there was nothing forcing the particle to choose which slit to go through, and so it could remain in the form of a probability wave.

This new experiment, therefore, demonstrates something fundamental in quantum physics: technically, it is not the act of measuring that influences the experiment, but whether or not this information exists somewhere in the universe. If this information is destroyed, the interference pattern returns.

The key is therefore information.

NOTE: How does the destruction of this information work? One might think it would simply be a matter of having the photon absorbed by an absorbing surface before reading it, but this does not work, and we are left with bands. Indeed, by doing so, the information theoretically exists because the absorbing surface could have determined the position of the particle through the orientation of the photon. The destruction works with another incredible principle of quantum physics that I will not detail in this article: entanglement. The photon is sent onto a special crystal, which splits it into two twin photons quantumly linked. One of the twins is then destroyed, making the information unrecoverable because to read the information, one absolutely needs to read both twins. To simplify, the two twins are not copies; they form a single system whose properties are not individually defined.

The Tunnel Effect

We are slowly getting closer to SSDs. But before that, there is one last quantum concept we need to talk about: the tunnel effect.

We said that an unobserved electron does not exist like a marble at a precise location. It exists as a probability wave spread out in space. This wave function gives a probability of finding the electron at each point in space.

Now let’s imagine a physical barrier. We send an electron toward this barrier. Classically, if the electron does not have enough energy to pass over it, it is blocked. Full stop.

Yet quantum mechanically, the wave function of the electron does not stop abruptly at the barrier. Because it is a wave, it propagates and gradually decays through the barrier. It does not fall to zero. On the other side, there therefore remains a non-zero probability of finding the electron.

This is the tunnel effect: a real chance for the electron to end up on the other side, without having had the classical energy needed to cross.

This probability is not fixed. It depends directly on the thickness of the barrier: the thinner the barrier, the more the wave function survives on the other side, and the higher the tunneling probability. At our scale, the barriers are far too thick for this effect to be observable. But at the scale of a few nanometers, the probability exists.

How SSDs Use Quantum Physics

In an SSD, we want to store data. They work with bits, but it is precisely in the management of these bits that the principles of quantum physics come into play.

In an SSD, each bit is encoded in cells called floating gates: small zones isolated on all sides by an insulating layer. This box can contain electrons or not:

• Box with electrons: Bit = 0

• Box without electrons: Bit = 1

If we need:

• To write, we therefore need to make electrons enter this isolated box. What we do is apply an electric voltage that deforms the wave function of the electrons and increases their probability of ending up on the other side. The electrons, therefore, cross the barrier via the tunnel effect.

• To erase, we apply a reverse voltage, which also impacts the wave function, and the electrons cross in the other direction.

• To read, it is a classical, non-quantum measurement: we measure the electric current passing through the transistor.

  • Electrons present: weak current: 0.

  • No electrons: strong current: 1.

We saw, however, that the wave function gives a probability, not a certainty. If we apply a voltage to write or erase, we therefore only have a probability that the electron will cross the barrier.

How can an SSD be reliable then? An individual electron is unpredictable, but we never send just one electron. We send millions simultaneously. Statistically, enough of them cross the barrier to charge the floating gate reliably.

And after each write, the controller immediately re-reads the cell to verify. If not enough electrons have crossed, it tries again. That is why SSDs embed error correction mechanisms, ECC (Error Correcting Code), precisely because the process is probabilistic by nature.

When a cell exceeds a certain error threshold over time, it is finally marked as defective and taken out of service. The data it held is moved to a healthy cell.

That is why SSDs always have an over-provisioning capacity: a reserve of cells invisible to the user, planned from the manufacturing stage to replace defective cells over time. And that is also why an SSD does not fail all at once; it degrades progressively, cell by cell, until the reserve is exhausted.

And this is where quantum physics imposes its limits. The more transistors shrink, the thinner the insulating barriers become, and the more the tunnel effect becomes uncontrollable, electrons escape spontaneously, errors increase, and cells age faster. Moore’s Law, which predicts a doubling of transistor density every two years, is today running up against these fundamental physical limits. This is not an engineering problem: it is quantum physics that sets the boundary.

Summary

• Matter is made up of atoms, themselves composed of a nucleus (protons and neutrons) and electrons. An atom is almost entirely empty: what we perceive as “solid” is an illusion created by the electromagnetic forces between atoms.

• Light is both an electromagnetic wave and a particle called a photon. In flight, it behaves like a wave, but it is emitted and absorbed like a particle, in one single hit, in one single place.

• Young’s double-slit experiment proves that light is a wave: it produces an interference pattern, impossible to obtain with classical particles.

• Matter behaves in the same way. But unlike light, its wave is not physical: it is a probability wave that describes the possible positions of a particle. This is quantum superposition: an unmeasured particle exists in multiple states simultaneously.

• It is not the act of measuring that collapses the superposition: it is the existence of the information somewhere in the universe. If the information is destroyed, the superposition is restored.

• Decoherence explains why we never see superposition at our scale: any macroscopic object permanently interacts with its environment, which instantaneously collapses its wave function.

• The tunnel effect is a direct consequence of the wave-like nature of particles: the wave function of an electron does not stop abruptly at a physical barrier. There exists a non-zero probability of finding it on the other side, without having had the classical energy to cross.

• SSDs exploit the tunnel effect to write and erase data: an electric voltage deforms the wave function of electrons and increases their probability of crossing the insulating barrier of a floating gate. Reliability rests on the large number of electrons sent and on ECC.

Resources

More From the Systems Category

• Instruction Pipelining

• Simultaneous Multithreading

• Linux Soft vs. Hard Lockup

Sources

• Something Deeply Hidden

• We Have No Idea

Explore Further

• Quantum Country

• The Double-Slit Experiment - Veritasium

❤️ If you enjoyed this post, please hit the like button.

💬 Did you know quantum physics was hiding in your laptop all along? I’d love to hear your reaction in the comments.

Leave a comment

Introduction

If you’ve been a subscriber since mid-2025, you were already here when I published the post that performed best on my newsletter: Working on Complex Systems.

I really loved writing it, and I always had in mind to revisit it at some point. So, when someone at ScyllaDB reached out to invite me to speak at Monster SCALE Summit, I saw the perfect opportunity to turn it into a talk.

The video isn’t a 1:1 mapping of the original content. I expanded it with more examples and new ideas. In it, I define what complex systems are, then discuss their common characteristics, and finally explore patterns for navigating them.

Hope you will enjoy it!

Video

Resources

More From the Distributed Systems Category

• Latency and User Experience

• Probabilistic Increment

• Bloom Filters

Explore Further

• Slides

• Monster SCALE Summit - 2026 Tech Talks

❤️ If you enjoyed this post, please hit the like button.

Leave a comment

At Google, I recently switched to a new domain: Google Distributed Cloud Connected1. Here, all the teams are very busy, and finding an efficient way to communicate over email or chat can be challenging, especially when asking someone to do something.

Recently, I came across a simple technique: three bullets and one call to action. The idea is the following:

• Add three bullet points explaining the key context

• Follow with one clear call to action

Let’s look at a concrete example. Suppose you receive the following email:

Hi,

I recently wrote a design doc on how to save storage in the context of X, where I describe the current problem and the approach we could take to address it. In the document, I go through the main trade-offs involved and explain why the proposal focuses on solution Y in particular. I also included several open questions related to the deployment strategy and some areas where feedback would be especially helpful. It would be great if you could take a look at the document and leave comments by Friday.

Thanks!

Quite a mouthful. It requires a non-trivial amount of brain time to understand both the context and what the person is actually asking for.

Now let’s apply the three bullets and a call to action strategy:

Hi,

• I recently wrote a design doc on how to save storage in the context of X.

• It highlights the main trade-offs and focuses on the solution Y.

• I’ve added open questions around the deployment strategy.

Could you please have a look and leave comments by Friday?

Thanks!

Much better, right? The call to action is clear, and the context is structured around short and easy-to-scan sentences.

Why does it work? When communicating via email or chat, people prefer short and memorable messages that do not require too much cognitive effort to process. Bullet points help break information into smaller chunks, which makes the message easier to scan quickly. Ideally, the bullet points and the call to action should be as short as possible.

Another aspect is that 3 is often a magic number in communication. With 2 items, you often get a contrast. With 3 items, you start to get a small structure or rhythm that is easier for the mind to process. That is one of the reasons why the rule of three appears so often in writing, storytelling, and presentations, where it helps make ideas more engaging and convincing.

Remember: to improve your chances of getting an answer to your request, use 3 short bullets and an efficient call to action.

Resources

More From the Beyond the Code Category

• 10 Rules I Learned About Technical Writing

• The XY Problem

• Don’t Forget About Your Mental Health

Sources

• Rule of three — Thinking Insights

• Rule of three (writing) — Wikipedia

Only 2, sorry about that.

❤️ If you enjoyed this post, please hit the like button.

💬 What do you think about this strategy? Have you tried something similar?

Leave a comment

Network Effects

Let’s consider a dating app. If there are only three people who installed the app in New-York, anyone new will probably try it for a few seconds and uninstall it. But if a big part of the city is on it, then someone single will probably stick around.

Said differently, the more people use the product, the more valuable it becomes. There’s a term for that type of product, and it’s called the network effect: when a product gets more valuable as more people use it.

Delving into the network effect, it’s not a single force but actually a trio of forces:

• The acquisition effect: How a product can use its own network to attract new people.

• The engagement effect: The more people join, the more useful and sticky it becomes.

• The economic effect: A larger network reduces costs, improves monetization, and strengthens the business model.

Now comes the real challenge. If our product relies on network effects, how do we launch it? Do we start from zero, or wait until enough people are on board? It’s a chicken-and-egg problem, and that’s the cold start problem.

Anti-Pattern: Big Bang

One common mistake to solve the cold start problem is the big bang launch: releasing to everyone before any community exists.

Google+ is the perfect example. Launched in 2011, it was Google’s attempt at a social network with a Facebook-style feed. The problem was that when people joined, they found empty timelines and left. Google later admitted that 90% of user sessions lasted less than five seconds.

At one point, Google even tied YouTube comments to Google+, requiring an account just to comment. The platform eventually reached more than 500 million users, but the issue was never sign-ups. The real problem was that a newcomer’s first session didn’t feel like walking into a lively room. It was forced growth instead of real networks.

Google+ didn’t fail because Google couldn’t build a social network. It lost because it never created a place where a new user could land in a live network. In short, Google+ failed to solve the cold start problem.

Pattern: Atomic Networks

A solution to the cold start problem was applied by Tinder, and it involves focusing on the concept of atomic networks.

Back then, dating apps were not very popular. Yet, Tinder was ambitious and wanted to succeed in that market. Instead of launching worldwide, they did the complete opposite. They organized a party at a college that required installing the application to attend. The next day, most students there had Tinder installed. This college was an atomic network: the smallest self-sustaining cluster of users where network effects actually work.

Soon after, they repeated the same process in another college in the same city, with the same result: within days, most students there had joined Tinder.

What’s powerful about atomic networks is simple: if we can build one network, we can build two. If we can build two, we can build thousands. Tinder repeated this strategy college by college, then city by city, eventually growing into entire countries.

The takeaway is that when we start a product with network effects, the first step is to build a single, tiny network that’s self-sustaining on its own. We just want to get started. If we can create one stable, engaged network, then we can build a second one next to it. From there, we can replicate the process and eventually connect them into one large network that spans the whole market.

Summary

• Network effects happen when a product gets more valuable as more people use it.

• How do we launch such a product with zero users? That’s the cold start problem.

• Big bang launches fail when newcomers find empty networks.

• One effective approach is to build atomic networks: the smallest self-sustaining clusters where network effects work.

• If we can build one atomic network, we can repeat it and scale across a market.

Resources

More From the Beyond Code Category

• Don’t Forget About Your Mental Health

• The XY Problem

• Lateral Thinking

Sources

• The Cold Start Problem

• Project Strobe: Protecting your data, improving our third-party APIs, and sunsetting consumer Google+

• Google+: Communities and photos

❤️ If you enjoyed the post, please consider giving it a like.

💬 The book is really great and covers many more aspects. I definitely recommend it. Are you into product management? What resources would you recommend?

Leave a comment

Join the Discord

☕ Welcome to The Coder Cafe! Today, we discuss a recent comment from Linus Torvalds about the use of a helper function. Get cozy, grab a coffee, and let’s begin!

In August 2025, there was (yet another) drama involving Linus Torvalds replying on a pull request:

No. This is garbage and it came in too late. I asked for early pull requests because I’m traveling, and if you can’t follow that rule, at least make the pull requests good.

This adds various garbage that isn’t RISC-V specific to generic header files.

And by “garbage” I really mean it. This is stuff that nobody should ever send me, never mind late in a merge window.

Like this crazy and pointless make_u32_from_two_u16() “helper”.

That thing makes the world actively a worse place to live. It’s useless garbage that makes any user incomprehensible, and actively WORSE than not using that stupid “helper”.

If you write the code out as “(a << 16) + b”, you know what it does and which is the high word. Maybe you need to add a cast to make sure that ‘b’ doesn’t have high bits that pollutes the end result, so maybe it’s not going to be exactly pretty, but it’s not going to be wrong and incomprehensible either.

In contrast, if you write make_u32_from_two_u16(a,b) you have not a f^%$ing clue what the word order is. IOW, you just made things WORSE, and you added that “helper” to a generic non-RISC-V file where people are apparently supposed to use it to make other code worse too.

So no. Things like this need to get bent. It does not go into generic header files, and it damn well does not happen late in the merge window.

Let’s not discuss the rudeness of this comment (it’s atrocious). Instead, let’s focus on the content itself.

, a popular newsletter, wrote a post about it:

the main point Linus makes here is that good code optimizes for reducing cognitive load. {…] Humans have limited working memory capacity - let’s say the human brain can only store 4-7 “chunks” at at time. Each abstraction or helper function costs a chunk slot. Each abstractions costs more tokens.

I share the view that good code optimizes for reducing cognitive load1, but I don’t understand Linus’s comment in exactly the same way.

Yes, Linus is virulent about the helper function, but in my opinion, his main argument isn’t simply that an abstraction costs a “chunk slot” as mentioned; it’s rather that this isn’t the right abstraction.

Here is the code added in the pull request:

/**
 * make_u32_from_two_u16 - return u32 number by combining
 * two u16 numbers.
 * @hi: upper 16 bit number
 * @lo: lower 16 bit number
 */
#define make_u32_from_two_u16(hi, lo)	(((u32)(hi) << 16) | (u32)(lo))

This macro builds a 32-bit integer by putting one 16-bit value in the high half and the other in the low half. For example:

hi     = 1010101111001101
lo     = 0001001000110100
result = 10101011110011010001001000110100
         |--------------||--------------|
               hi               lo

The main problem with this macro isn’t necessarily that it exists. It’s that its intent (meaning what it tries to accomplish) could have been clearer.

Indeed, the helper’s name doesn’t tell which word is high and which one is low and that’s exactly what Linus is calling out with “you have not a f^%$ing clue what the word order is”.

Because we can’t get the intent from the name (make_u32_from_two_u16), we have to open the macro to understand the order. That’s precisely why it costs a “chunk slot.”: not because the abstraction exists, but because it’s an ambiguous one.

If we wanted to keep using a macro, a better approach, in my opinion2, would be to encode the word order in the name itself (msw = most significant word, lsw = least significant word):

/**
 * make_u32_from_msw_lsw - return u32 number by combining
 * two u16 numbers in msw:lsw word order.
 * @msw: upper 16 bit number
 * @lsw lower 16 bit number
 */
#define make_u32_from_msw_lsw(msw, lsw) (((u32)(msw) << 16) | (u32)(lsw))

In this case, the word order is carried by the macro name, which makes it a clearer abstraction. Reading the call site doesn’t require opening the macro to understand the word order:

u32 v = make_u32_from_msw_lsw(0xABCD, 0x1234);

Such an abstraction doesn’t cost a “chunk slot” in terms of cognitive load. Its intent is clear from the name, so we don’t need to load an extra piece of information into our working memory to understand it.

In summary, if we want to optimize for cognitive load, there’s not necessarily an issue with using helper functions. But if we do, we should make the abstraction as explicit as possible, and that starts with a clear function name that conveys what it tries to accomplish.

Resources

More From the Programming Category

• Readability

• Cognitive Load

• Nested Code

Sources

• Re: [GIT PULL] RISC-V Patches for the 6.17 Merge Window, Part 1 - Linus Torvalds // The discussion.

• GitHub // The code proposed in the pull request

Explore Further

• Linus and the two youts // Interestingly, the macro was plain wrong when the second word was negative. The full explanation is here.

❤️ If you enjoyed this post, please hit the like button.

💬 Where do you draw the line between “helpful” and “harmful” abstraction?

Leave a comment

☕ Welcome to The Coder Cafe! Today, November 13, is World Kindness Day. For this special occasion, we discuss how kindness matters at work. Get cozy, grab a coffee, and let’s begin!

We’re in 2022. It’s Saturday evening, and I’m about to go to bed. I’m on-call that night. I haven’t been paged, but just to make sure everything is OK, I logged in and checked Slack.

An incident was going on, and a colleague was already on it. I DMed him: “Why didn’t you contact me?”

He replied: “It’s late and I thought you might be sleeping. I was awake, so I looked to see if there’s something I could do.”

My first reaction was: I’m on-call. I’m paid for it. I’ll take care of it. Go to bed.

But here’s the thing: on a Saturday evening, he chose to help because he thought I might be sleeping, even though I was the one on-call, the one paid to handle it.

That was a pure act of kindness. No points. No credit. Just care. And after that? Honestly, I would have done anything for that person.

Why Kindness Wins

At work, we work with people long before we work with code. There’s always a little distance between us: roles and power dynamics, deadlines and pressure, different cultures, communication styles, sometimes different time zones. Kindness is the fastest bridge across that distance.

Kindness is about being generous, considerate, and having concern for others without expecting praise or reward in return. It’s a voluntary act that creates psychological safety among team members. When people feel safe, they surface risks earlier, ask the “naive” questions, and move faster together.

Kind people make work better day in, day out. Kindness boosts trust, speeds decisions, reduces stress, and quietly raises the bar for everyone.

Let’s look at a few places where kindness matters in our daily jobs:

• Code review: When we’re assigned a review, we’re not there to rate someone’s code. We’re there to merge the best possible change together. Be respectful and stay factual. Favor questions over pronouncements: “What scenarios does this handle? I’m worried about X; would Y cover Z?” Point out what’s good, suggest concrete fixes, and link to standards or examples. If there’s confusion, offer help.

• Meetings: Make space so everyone can be heard. Don’t interrupt. Invite quieter people in: “Ben, anything you would add?” It’s not because someone is more vocal that they’re more right.

• Mentoring: People make mistakes. Don’t jump to blame or perform expertise. The goal is to protect in public and correct in private. Give clear, kind feedback, focus on the next step, and share your own past mistakes to lower the temperature.

• Random thank you: When you receive help or just enjoy working with someone, say thank you. Recognition matters, and doing it publicly multiplies the effect. For example, at Google, there’s a program called gThanks that lets you thank someone publicly so others can see it too.

• Make time to listen: Being kind also means making time to listen. I remember going through a difficult period of my life, and a former manager just took time to talk, without judging. That mattered more than any advice.

• Self-compassion: Kindness also applies to yourself. Give yourself the same understanding you would give a teammate. Take breaks, ask for help, forgive your own mistakes, and learn from them.

Being kind is a bridge to people, and even in a professional context, as Aesop wrote, no act of kindness, no matter how small, is ever wasted.

Resources

More From the Beyond the Code Category

• Don’t Forget About Your Mental Health

• Keeping a Mistake Journal

• The XY Problem

Sources

• Why Kindness at Work Pays Off

Explore Further

• Kind Engineering

❤️ If you enjoyed this post, please hit the like button.

💬 What’s one act of kindness that changed your workday?

Leave a comment

🎃 Welcome to The Coder Cafe! Today, we examine the Therac-25 accidents, where design and software failures resulted in multiple radiation overdoses and deaths. Make sure to check the Explore Further section to see if you’re able to reproduce the deadly issue. Get cozy, grab a pumpkin spice latte, and let’s begin!

Therac-25

Treating cancers used to require a mix of machines, depending on tumor depth: shallow or deep. In the early 1980s, a new generation promised both from a single system. That was a big deal for hospitals: one machine instead of several meant lower maintenance and fewer systems to manage.

That was the case with the Therac-25.

The Therac-25 offered two therapies with selectable modes:

• Electron beam: Low-energy electrons for shallow tumors (e.g., skin cancer).

• X-ray photons: High-energy radiation for deep tumors (e.g., lung cancer).

Earlier Therac models allowed switching modes with hardware circuits and physical interlocks. The new version was smaller, cheaper, and computer-controlled. Less hardware and fewer parts meant lower costs.

However, what no one realized soon enough: it also removed an independent safety net.

The Horror Begins

On a routine day, a radiology technologist sat at the console and began entering a plan:

• By habit, she selected X-ray (deep mode).

• Then she immediately corrected it for Electron (shallow mode) and hit start.

The machine halted with a MALFUNCTION 54 message. The operator’s manual didn’t explain the code. Service materials listed the number but gave no useful guidance, so she resumed and triggered the radiation.

The patient was receiving his ninth treatment. Immediately, he knew something was different. He reported a buzzing sound, later recognized as the accelerator pouring out radiation at maximum. The pain came fast; paralysis followed. He later died from radiation injury.

Weeks later, a second patient endured the same incident on the same model.

What Happened?

Initially, the radiology technologist entered ‘X’ for X-ray (‘▮’ is the cursor and ‘…’ are other fields):

MODE:              X
...                ▮
BEAM READY:        

She immediately hit Cursor Up to go back and correct the field to ‘E’:

MODE:              E
...                ▮
BEAM READY:        

After a rapid sequence of Return presses, she moved back down to the command area:

MODE:              E
...                
BEAM READY:        ▮

From her perspective, the screen showed the corrected mode, so she hit return and started the treatment:

MODE:              E
...                
BEAM READY:        ▮<Enter>

Behind the scenes, the Therac-25 software ran several concurrent tasks:

• Data-entry task: Monitored operator inputs and edited a shared treatment-setup structure.

• Hardware-control task: On a periodic loop, snapshotted that same structure and positioned the turntable and magnets based on user input.

Because both tasks read the same memory with no mutual exclusion, there was a short window (on the order of seconds) in which the hardware-control task used a different value than the one displayed on the screen.

As a result:

• The UI showed Electron mode, which looked correct to the operator.

• The hardware-control task had snapshotted stale data and marked the system as ready even though critical elements (e.g., turntable position, scanning magnets/accessories) were not yet aligned with electron mode.

• When treatment was started, the machine delivered an effectively unscanned, high-intensity electron beam, causing a massive overdose.

This is a race condition example: the outcome depends on the timing of events, here, the input cadence of the technologist. Depending on the timing, the system could enter a fatal state, with one process seeing ‘X’ while another saw ‘E’.

The manufacturer later confirmed the error could not be reproduced reliably in testing. The timing had to line up just right, which made the bug elusive. They initially misdiagnosed it as a hardware fault and applied only minor fixes. Unfortunately, the speed of operator editing was the key trigger that exposed this software race.

Another Tragedy

The problem could have stopped here, but it didn’t.

Months later, another fatal overdose occurred, this time caused by a different software defect. It wasn’t a timing race. This time, the issue was a counter overflow within the control program.

The software used an internal counter to track how many times certain setup operations ran. After the counter exceeded its maximum value, it wrapped back to zero. That arithmetic overflow created a window where a critical safety check was bypassed, allowing the beam to turn on without the proper accessories in place.

Again, the Therac-25 fired a high-intensity beam without the proper hardware configuration.

Conclusion

Both the race condition and the counter overflow stemmed from the same design flaw: the belief that software alone could enforce safety. The Therac-25 showed, in tragic terms, that without independent safeguards, small coding errors can have catastrophic consequences.

We should know that whether it’s software, hardware, or a human process, every single safeguard has inherent flaws. Therefore, in complex systems, safety should be layered, as illustrated by the Swiss cheese model:

In total, there were six known radiation overdoses involving the Therac-25, and at least three were fatal.

Resources

More From the Reliability Category

• Adaptive LIFO

• Resilient, Fault-tolerant, Robust, or Reliable?

• Lurking Variables

Sources

• The Worst Computer Bugs in History: Race conditions in Therac-25

• Killed By A Machine: The Therac-25

• An Investigation of the Therac-25 Accidents

Explore Further

I created a Docker image based on a C implementation from an MIT course simulating the operator console of the Therac-25 interface:

You can run the UI using Docker:

 docker run --rm -it -e TERM=xterm-256color teivah/therac-25

Simulator commands:

• Beam Type: ‘X’ or ‘E’

• Command: ‘B’ for beam on, or ‘Q’ to quit the simulator.

👉 Try to trigger the MALFUNCTION 54 error based on the scenario discussed.

❤️ If you enjoyed this post, please hit the like button.

💬 Any other horror coding stories you want to share?

Leave a comment

We often celebrate teams for moving fast. But speed alone can be a trap. A rush of fast changes that barely move the product toward the real goal shouldn’t count as a win. When we talk about team productivity, we should understand that speed ≠ velocity:

• Speed is how quickly a team ship changes.

• Velocity is speed with direction, the movement toward a defined goal.

Let’s look at three teams to illustrate these definitions.

• Team A has ideal velocity. Each iteration is represented by an arrow: its length shows speed, and its angle shows direction. Every iteration moves the team consistently closer to the goal.

  

• Team B has a speed problem. Each iteration is well-aligned (correct angle), but the team delivers too slowly (small length), resulting in a lower velocity. They finish only halfway to the goal.

  

• Team C ships as rapidly as team A (same arrow length). However, various factors such as frequent bug fixes and changing targets make their direction inconsistent. Despite high speed, their velocity is low, and they end only halfway to the goal, just like team B.

  

As we can see, velocity requires both speed and direction. A team moving too slowly or in inconsistent directions will make little progress, even if they’re busy. Only when speed is high and direction is aligned do teams reach their goals efficiently.

Measuring team speed isn’t useless, though. We can track speed by considering various metrics such as deployment frequency, average time in code review, or mean time to recovery (MTTR) following a production bug. These metrics are interesting to track and provide a certain perspective to understand a team's productivity. Yet, speed shouldn’t be the sole dimension to track.

The danger of tracking speed only is that a team might become organized in a way to optimize short-term delivery. The team might focus on delivering many changes that, together, do not move the product in a meaningful way. Instead, teams should track speed and velocity.

As we said, velocity is speed with direction. We already discussed metrics to track speed; what is missing is monitoring direction.

Setting clear and factual objectives that align with the business strategy helps us track direction. For example:

• Payment success rate above 99 percent.

• Signup to activation above 50 percent within 7 days.

• Retention after week 4 is above 40 percent.

The easier a metric is to measure, the easier it is to track the direction over time.

One caveat is how to report progress. In a past team, we used to set OKRs (Objectives and Key Results) every semester. Some objectives were difficult to measure, so we tracked progress differently. Say we created 50 tickets and closed 45. In that case, we reported that we reached 90% of the OKR. That number said nothing about real progress toward the objective. Key results should be outcomes, not ticket counts.

Something else to mention, I read here and there that we should track velocity by counting the number of story points delivered within a timeframe (e.g., during a sprint). I strongly disagree with this. Let me give an example:

• A team ships a first change ← 3 story points

• The team finds a bug and ships a fix ← 2 story points

• Later, the team learns the initial approach is not the best, so it ships another change ← 5 story points

On paper, the team delivered 10 story points. That is only speed, though, not velocity. If we care about direction as well, the team only delivered a single feature. Story points measure effort; they don’t measure progress toward the objective.

Summary

• Speed is how fast we ship. Velocity is speed with direction toward a goal.

• The goal of a team shouldn’t be to reach high speed; it should be to reach high velocity, where rapid iterations translate into real system-level improvements.

• Tracking story points, for example, is a measure of speed, not velocity.

• Objectives tracking should use outcomes, not ticket counts. Reporting 90 percent of tickets done is not a good measure of progress toward the objective.

Resources

More From the Beyond Code Category

• Keeping a Mistake Journal

• Streetlight Effect

• Survivor Bias

Sources

• Sprint Velocity in Scrum: How to Measure and Improve Performance // The velocity definition I disagree with.

❤️ If you enjoyed this post, please hit the like button.

💬 What’s your take on speed vs. velocity? How do you measure velocity in your team?

Leave a comment

CRDTs, short for Conflict-Free Replicated Data Types, are a family of data structures built for distributed systems. At first sight, CRDTs may look intimidating. Yet at their core, the idea is not that complex. What makes them special is that they allow updates to happen independently on different nodes while still guaranteeing that all replicas eventually converge to the same state.

To understand how CRDTs achieve this, we first need to step back. We need to talk about concurrent operations and what coordination means in a distributed system. Let’s take it step by step.

Concurrent Operations

What does concurrent operations mean? Our first intuition might be to say they happen at the same time. That’s not quite right. Here’s a counterargument based on a collaborative editing example.

1. While on a plane, Alice connects to a document and makes an offline change to a sentence.

2. An hour later, Bob connects to the same document and edits the very same sentence, but online.

3. Later, when Alice lands, both versions have to sync.

The two edits (1. and 2.) were separated by an hour. They didn’t happen at the same time, yet they are concurrent.

So what’s a better definition for concurrent operations? Two operations that are not causally related.

In the previous example, neither operation was made with knowledge of the other. They are not causally related, which makes them concurrent. Yet, if Bob had first seen Alice’s update and then made his own, his edit would depend on hers. In that case, the two operations wouldn’t be concurrent anymore.

We should also understand concurrent ≠ conflict:

• If Alice fixes a missing letter in a word while Bob removes the whole word, that’s a conflict.

• If Alice edits one sentence while Bob edits another, that’s not a conflict.

Concurrency is about independence in knowledge. Conflict is about whether the effects of operations collide.

Coordination

Now, let’s talk about coordination in distributed systems.

Imagine a database with two nodes, node 1 and node 2. A bunch of clients connect to it. Sometimes requests go to node 1, sometimes to node 2. Let’s say two clients send concurrent and conflicting operations:

In this case, we can’t have node 1 storing $200 while node 2 stores -$100. That would be a consistency violation with the two nodes disagreeing on Alice’s balance.

Instead, both nodes need to agree on a shared value. To do that, they have to communicate and decide on one of the following:

• Reject both operations

• Accept client A’s update and set the balance to $200

• Accept client B’s update and set the balance to -$100

The very action of nodes communicating and, if needed, waiting to agree on a single outcome is called coordination.

Coordination is one way to keep replicas consistent under concurrent operations. But coordination is not the only way. That’s where CRDTs come in.

CRDT

CRDT stands for Conflict-Free Replicated Data Types. In short, CRDTs are data structures built so that nodes can accept local updates independently and concurrently, without the need for coordination.

If you read our recent post on availability models, you might notice we’re now in the territory of total availability: a system is totally available if every non-faulty node can execute any operation. Total availability comes with weaker consistency. For CRDTs, the consistency guarantee is called Strong Eventual Consistency (SEC).

For that, CRDTs rely on a deterministic conflict resolution algorithm. Because every node applies the same rules, all replicas are guaranteed to eventually converge to the same state.

G-Counter

Let’s make this more concrete with a classic CRDT: the G-Counter (Grow-Only Counter).

Imagine a database with two nodes tracking the number of likes on a post. Node 1 receives a new like, increments its counter, and replies success to the client:

Then, node 1 communicates with node 2 to send this update:

Ultimately, both nodes converge to the same value: 6.

How does the conflict resolution work for a G-Counter?

Each replica keeps a vector of counters, with one slot per node. In our example, the total number of likes is 5. Let’s say node 1 has seen 2 likes and node 2 has seen 3 likes. So the initial state is the following:

Node 1: [2, 3]
Node 2: [2, 3]

When node 1 receives a new like, it only increments its own slot. Node 2 is now temporarily out of sync:

Node 1: [3, 3]
Node 2: [2, 3]

During synchronization, both nodes merge their vectors by taking the element-wise maximum:

[max(3,2), max(3,3)] = [3, 3]

Now both replicas converge to the same state:

Node 1: [3, 3]
Node 2: [3, 3]

The beauty of this algorithm is that it’s deterministic and order-independent. No matter when or how often the nodes sync, they always end up with the same state.

NOTE: Do you know Gossip Glomers? It’s a series of distributed systems challenges we briefly introduced in an earlier post. Challenge 4 is to build a Grow-Only Counter. It’s worth checking out if you haven’t already.

PN-Counter

CRDTs can also be combined to make a more complex CRDT. For example, if we want to track both likes and dislikes, we can use two G-Counters together. This data type is called a PN-Counter (Positive-Negative Counter).

Imagine two clients act concurrently on the same post: one likes it, another dislikes it.

The nodes exchange their updates and converge to the same value:

In the case of a PN-Counter, the conflict resolution algorithm is similar to the G-Counter. The difference lies in the fact that it involves not one but two vectors: one for increases and one for decreases.

Assume an initial state where node 1 has received 2 likes and 0 dislikes, and node 2 has received 3 likes and 0 dislikes:

-- Increase
Node 1: [2, 3]
Node 2: [2, 3]

-- Decrease
Node 1: [0, 0]
Node 2: [0, 0]

Now, suppose node 1 receives a new like and node 2 receives a dislike. Before the sync, the state is the following:

-- Increase
Node 1: [3, 3]
Node 2: [2, 3]

-- Decrease
Node 1: [0, 0]
Node 2: [0, 1]

When the replicas exchange their state, the merge rule is element-wise maximum for each vector:

-- Increase
[max(3,2), max(3,3)] = [3, 3]

-- Decrease
[max(0,0), max(0,1)] = [0, 1]

After sync, both nodes converge to:

-- Increase
Node 1: [3, 3]
Node 2: [3, 3]

-- Decrease
Node 1: [0, 1]
Node 2: [0, 1]

The final counter of likes is:

node1Likes + node2Likes - node1Dislikes - node2Dislikes
= 3 + 3 - 0 - 1
= 5

Use Cases

Let’s pause for a second. Based on what we’ve discussed, can you think of some use cases for CRDTs? A data structure where nodes are updated independently, concurrently, without coordination, and still guarantees that they converge to the same state?

One main use case is collaborative and offline-first systems. For example, Notion, a collaborative workspace, recently introduced a feature that lets people edit the same content offline. They rely on CRDTs, and more specifically on Peritext, a CRDT for rich-text collaboration co-authored by multiple people, including .

Another big use case is totally available systems that put availability ahead of strong consistency. As we’ve seen, nodes don’t need to coordinate before acknowledging a client request, which makes the system more highly available.

Take Redis, for example. It can be configured in an active-active architecture with geographically distributed datacenters. Clients connect to their closest cluster and get local latencies without waiting for coordination across distant regions. And yes, this setup is built on CRDTs.

We could also think about other applications for CRDTs, like:

• Edge & IoT: Devices update offline and merge later without a central server.

• Peer-to-peer: Peers share changes directly and match up when they reconnect.

• CDN/edge state: Keep preferences, drafts, or counters near users and sync to the origin later.

Different Types of CRDTs

There are two main types of CRDTs:

• State-based CRDTs: Convergence happens by propagating the full state.

• Operation-based CRDTs: Convergence happens by propagating the update operations.

State-Based CRDTs

In the previous examples, we looked at two state-based CRDTs: the G-Counter (Grow-Only Counter) and the PN-Counter (Positive-Negative Counter). In both cases, what was exchanged between the nodes was the entire state. For example, node 1 could tell node 2 that its total number of likes is 3.

With state-based CRDTs, states are merged with a function that must be:

• Commutative: We can merge in any order and get the same result.

• Idempotent: Merging something with itself doesn’t change it.

• Associative: We can merge in any grouping and get the same result.

Each synchronization monotonically increases the internal state. In other words, when two replicas sync, the state can only move forward, never backward. This is enforced by a simple “can’t-go-backwards” rule (a partial order), where merges use operations like max for numbers (as we’ve seen) or union for sets.

Operation-Based CRDTs

In operation-based CRDTs, nodes share the operations rather than the full state. Convergence relies on three properties:

• Commutativity of concurrent operations

• Causality: Either carried in the operations’ metadata (for example, vector clocks) or guaranteed by the transport layer through causal delivery

• Duplicate tolerance: Handled by idempotent operations, unique operation IDs with deduplication, or a transport layer that guarantees no duplicates

One example of an operation-based CRDT is the LWW-Register (Last-Writer-Wins Register), which stores a single value. Updates are resolved using a logical timestamp (such as Lamport clocks) along with a tie-breaker like the node ID. When a node writes a value, it broadcasts an operation (value, timestamp, nodeID). On receiving it, a node applies the update if the pair (timestamp, nodeID) is greater than the one it currently holds.

CRDT Types Summary

To summarize:

• State-based CRDTs:

  • Convergence is guaranteed because merging states is associative, commutative, and idempotent.

  • Don’t require assumptions on the delivery layer beyond eventual delivery.

  • Simpler to reason about.

  • Exchanging full states can be more bandwidth-intensive.

• Operation-based CRDTs:

  • More bandwidth-efficient; we only send the operations, not the whole state.

  • Correctness usually depends on having causal order (or encoding causality in the ops) and tolerating duplicates via idempotence/dedup.

  • More complex to implement (causal broadcast, vector clocks, or equivalent).

For completeness, there’s also a third type we should be aware of: delta-based CRDTs. Here, convergence is achieved by sending and merging fragments of state (deltas) rather than the entire state. A quick analogy to picture the differences:

• State-based CRDT: “From time to time, send me the whole document.”

• Operation-based CRDT: “When you make a change, tell me exactly what you did.” → “Adding word `miles` at position 42.”

• Delta-based CRDT: “When you make a change, send me just the delta that reflects it (for example, the updated sentence)” → “And miles to go before I sleep.”

CRDT vs. OT

We talked about collaborative document editing. So you might assume a system like Google Docs is based on CRDTs, right?

Well, that’s not the case. Google Docs is based on another concept called OT (Operational Transformation). The goal of OT and CRDT is the same: convergence among all nodes in a collaborative system.

The main difference is that OT requires all communication to go through the same server:

We haven’t mentioned it until now (on purpose), but with CRDTs, there’s no need for a central server to achieve convergence.

Back to our collaborative editing tool: if Alice and Bob are both offline but manage to connect their laptops directly, they could still achieve convergence without talking to a central server:

As we saw earlier, CRDTs embed a deterministic conflict resolution algorithm. The data type itself ensures convergence.

That’s the key difference: CRDTs don’t need to make any assumptions about the network topology or about a central server. considers CRDT to be the natural successor of OT.

NOTE: So, why is Google Docs still based on OT? Historical reasons. Google Docs was launched before CRDTs existed, and it still works really well. There’s no practical reason for Google to migrate from OT to CRDT, despite some discussions about it in the past.

Conclusion

• Operations are concurrent when they aren’t causally related; concurrency doesn’t automatically mean conflict.

• Coordination is when replicas communicate and, if needed, wait to agree on a single outcome for concurrent updates before acknowledging clients, so they don’t diverge.

• CRDTs accept independent updates on each replica and still converge via deterministic merge rules.

• Three types: state-based (share full state), operation-based (share operations), delta-based (share just the changed parts).

• CRDTs are a great fit for systems like offline-first collaboration and highly available systems.

• Unlike OT, CRDTs don’t rely on a central server to reach the same result everywhere.

Resources

More From the Distributed Systems Category

• Exploring Database Isolation Levels

• Consensus

• Safety and Liveness

Sources

• Ivan Zhao (Notion’s CEO) tweet on the new Notion offline collaboration feature

• Diving into Conflict-Free Replicated Data Types (CRDTs) - Redis

• CRDTs: The Hard Parts by

Explore Further

• Hacker News discussion

• Peritext - A CRDT for Rich-Text Collaboration

• Active-Active geo-distribution (CRDTS-based) - Redis

• Bartosz Sypytkowski’s 12-part blog series on CRDT

• crdt.tech

❤️ If you enjoyed this post, please hit the like button.

💬 Have you worked with CRDTs before, or do you see another use case where they shine? Share your thoughts in the comments!

Leave a comment

A few months ago, I was in a meeting discussing the state of a codebase and heard myself say:

After reflecting on my own words, I began questioning myself: I don’t even know what organic growth really means and whether it is ultimately positive or negative.

Discussing the concept with various colleagues, I noticed we had different interpretations. I also created a poll on X that went viral (#irony), showing that most people see organic growth as something positive.

If Perplexity is correct, the term organic growth was used first in Bushido: The Soul of Japan, published in 1899. The author, Inazo Nitobe, describes Bushido (aka the way of the samurai) as something that developed through organic growth rather than being the invention of a single person or the result of a single event. In other words, Bushido emerged gradually over decades and centuries as a collective experience, not the product of one individual.

It’s easy to draw parallels with the many codebases we maintain at work. The older the codebase and the more developers it has had, each with their own vision of how the code should look. Differences in perception arise from factors like developers’ varied mental models based on the parts of the code they know best.

Developers come and go, making changes, introducing biases about what the future should be, and after years, the codebase reflects hundreds or even thousands of such incremental changes.

This is organic growth in software: incremental delivery driven by agile methodology that shapes codebases, often in a bottom-up manner.

So, is organic growth good or bad? Let’s use the garden metaphor: imagine a public garden where anyone can plant whatever they want and wherever they want. Over time, sure, it’s colorful, but there’s little harmony, with a lack of overall guidance or direction:

Now imagine we control what can be planted and where, and periodically review and rearrange what’s growing. Our garden will likely look harmonious and well-maintained:

In my opinion, organic growth carries a somewhat negative connotation, implying a lack of direction, with systems growing out of ad hoc solutions and quick fixes rather than intentional design.

So, how can we move from a messy garden to a harmonious one? By transitioning from organic to controlled growth.

Controlled growth is the deliberate and reflective process of evolving a codebase. With controlled growth, we should also embrace progress that is not always perfect but consistently guided by shared patterns and standards, fostering sustainable, manageable, and harmonious development over time.

Here are five main rules for controlled growth:

• Be consistent. Follow agreed conventions and standards to build maintainable and predictable codebases. Many standards aren’t explicit, so strive to follow even the implicit ones. For example, if an entity is consistently named X, don’t be a cowboy and name it Y; stick to X.

• Plan ahead for large changes. Think strategically and collectively about big transitions such as significant new features. Avoid chaos and technical debt by breaking ambitious changes into smaller phases, communicating plans early, and leveraging documentation assets such as design docs (see Explore Further section).

• Hold regular retrospectives. After major development milestones, gather the team to review what went well and what could improve. Retrospectives foster a culture of learning and continuous improvement, a strategic pillar of controlled growth.

• Apply the Boy Scout rule. Frequent changes are inevitable and often rushed under pressure. Remember: “Always leave the campground cleaner than you found it.” When making a pull request, spend a little time refactoring small bits, cleaning legacy code, removing dead code, or renaming variables. This keeps the garden harmonious over time.

• Build a team where everyone shares responsibility and speaks openly. Controlled growth shouldn’t fall only on tech leads; it should be everyone’s job. When all team members feel heard and share responsibility for the health and quality of the codebase, the team grows on a stronger, more sustainable foundation.

Resources

More From the Programming Category

• Tidy First? // We already discussed organic growth in this post.

• Focus on Product Ideas, Not Requirements

• Cognitive Load

Sources

• Bushido: The Soul of Japan

Explore Further

• Design Docs at Google

❤️ If you enjoyed this post, please hit the like button.

💬 When you say a codebase grew organically, what do you mean?

Leave a comment

A Decade of Failures

For over a decade, I failed miserably at keeping an effective note-taking system, mostly for two reasons:

1. I never had a single, centralized place. I used to rely on a combination of Notion + Apple Notes + a physical notebook + Kindle highlights + Anki.

2. I didn’t have a system that was generic enough. I had a different format pretty much every time: book summaries, posts, computer science notes, mistake journal, personal growth notes, and so on.

The result was a messy system that didn’t scale. I kept losing knowledge, and it made learning inefficient. When we read or watch a great resource but don’t capture what we learned from it, chances are high that within a few months or even weeks, that knowledge fades away. That’s what kept happening to me.

In a world full of content—social media, work, books, courses, podcasts—being able to extract and retain working knowledge isn’t optional; it’s necessary if we want to keep growing as an engineer and as a person.

Let’s discuss an approach to solve this problem.

Second Brain

One of the first sentences in the book hooked me:

Our brain is great for processing ongoing tasks, but it’s not meant to retain everything. Over time, it lets go of unused ideas to make room for more relevant ones. This process, referred to as synaptic pruning, helps us adapt, but it also means we lose what we don’t externalize.

That’s the promise of a second brain: a place to offload notes and thoughts, avoid losing knowledge, and build on it over time with new ideas.

Before switching to the second brain approach, I also struggled with my notes because I wanted everything to be perfect. That drained a lot of time and energy. But one paragraph from the book really shifted my perspective:

That changed everything for me. I stopped chasing perfection and started building something that simply works, something that supports me every day, both at work and in my personal life.

What I found enriching is that the book presents not just a system, but also a mindset. Let’s start with the latter.

Mindset

Knowledge work is about taking information and turning it into results, for example, delivering on a project. All day, we consume and then produce:

The problem with this approach is that most of the information that we will gain will eventually be lost. Sure, we might remember the most important parts for a while. But what about the rest? It will fade, eventually.

What we miss is a feedback loop, a way to recycle information into knowledge that we can reuse later:

That’s what a second brain gives us: a way to turn information into assets we can reinvest in the future.

Another interesting idea in the book is to stop seeing notes as a flat list of things we’ve saved. Our notes can become building blocks, meaning pieces that help us create new ideas later on.

That’s a powerful shift: notes aren’t just storage, they’re raw material for thinking. They can connect and evolve into new ideas. Being able to track down all your notes in a single place makes those connections easier to spot and build on.

One last point I wanted to discuss: creativity. Here’s the definition given by Neuroscientist Nancy C. Andreasen:

A second brain is not only a memory tool, but also a thinking tool. Keeping an effective system to track down notes and ideas becomes a great tool to improve our creativity. Remember: creativity is not a talent, it’s a way of operating.

NOTE: Did you enjoy this punchline? It’s coming from a note I hold on creativity in my second brain 😎.

System

In this section, we’ll delve into what a note really is, how to take notes effectively, and then go over the PARA system presented in the book.

Notes

So, what’s a note? The author defines it this way:

Let’s go over the different parts of that definition.

First, a note is something to use, not just to collect. Again, we’re not building an encyclopaedia, we’re building something that works for us. For example, if we’re interested in public speaking, we don’t need a note on every single resource we read or watch.

Instead, the author suggests asking a few questions to decide what’s worth capturing:

1. Does it inspire us?

2. Is it useful?

3. Is this personal?

4. Is it surprising?

Ultimately, we should capture what resonates.

For example, we may have read this amazing book that everyone is talking about and discover that it doesn’t resonate with us at all1. Conversely, we may have watched a 30-second video that had a profound effect on us and triggered emotions. In this case, it’s better to spend some time creating a note on this video rather than the book.

When something resonates with us, it’s our emotion-based, intuitive mind telling us it’s interesting before our logical mind can explain why.

How to Take Notes

Every time we take a note, we should ask: “How can I make this as useful as possible for my future self?”

One way to do that is to be mindful of our future limited time. Instead of tracking dozens, if not hundreds, of lines, we should focus on finding the essence, meaning the heart and soul of what a resource is trying to communicate.

But in some cases, that’s barely possible. For example, I recently read the F1: A distributed SQL database that scales whitepaper. How can we capture the essence of such a dense technical document containing so much valuable information?

The solution, brought again by the author, is to use the progressive summarization technique. In short, it’s about layering our notes:

• Layer 1 - Captured notes: Either a copy and paste or even better writing, down in our own words what we understood from it.

• Layer 2 - Bolded passages: Go over the captured notes and mark in bold the most important pieces.

• Layer 3 - Highlighted passages: Go over the bolded passages and highlight the most important pieces.

• Layer 4 - Summary: Write down a summary.

Here’s an example from my F1 whitepaper notes. I created two main sections: Summary and Highlights.

• Layer 1: In Highlights, I captured all the raw content, for example, on locking.

  

• Layer 2: I bolded the passages that were most interesting to me.

  

• Layer 3: I highlighted the most important parts.

  

• Layer 4: I wrote a summary section with the ideas that were the most interesting for me.

  

In the end, the note looks like this:

# Summary

- ...
- ...
- ...

# Highlights

## Architecture

## Data model

## Locking

## Etc.

This approach allows me to come back later and start with a quick summary to refresh the main ideas. If I need a bit more, I can scan the bolded and highlighted passages. And if I really want to dive deeper, I can go through all the highlights.

Of course, every layer is optional. If we consider that a resource only needs layers 1 and 4 of summarization, or just 1 and 2, that’s perfectly fine.

Again, our second brain should be something that works for us.

PARA

The PARA system is at the core of the book. It’s a proposition on how to organize our notes. It is designed for actionability, with layers of action. To make things clear, a note can be assigned to one of the following domains:

• Project: A short-term effort with a possible due date and a clear outcome that needs to happen in order to mark the project as complete. For example, publish a blog post about second brain.

• Area: Ongoing responsibilities, what we are committed to, and what requires constant attention.

• Resource: A catchall for anything that doesn’t belong to a project or an area.

• Archive: When a note becomes inactive or outdated, we can move it to the archive.

NOTE: The distinction between area and resource wasn’t immediately clear to me, so here’s how I think about it.

I enjoy both fitness and climbing. But I’m only committed to fitness. I try to eat healthy, work out regularly, and so on. Climbing, on the other hand, is something I enjoy, but I only go from time to time. So in my system, fitness is an area, and climbing is a resource.

The PARA system has two main benefits.

• Clear focus: We’re not mixing short-term efforts with long-term maintenance. It helps us focus on outcomes and next steps rather than just piles of information.

• Genericity: PARA can handle all kinds of notes. It organizes information based on how actionable it is, not what kind of information it is.

As I said, I used to lack a system that was generic enough to track all my notes. Now, whether it’s a book I read, a course I followed, or a post I came across, I capture everything that resonates with me using the PARA system.

Notion

To implement the second brain, I used Notion, which I think is a fantastic tool with a lot of flexibility and configuration options.

If you don’t want to build your own second brain from scratch, you can check out this online tutorial:

Or, you can use my personal Notion template. This is the setup I use every day to track notes, growth tasks, my to-do list, areas, resources, and more.

Conclusion

In summary, the second brain method presented in the book follows the CODE system:

• Capture: Keep what resonates, leave the rest aside.

• Organize: Save for actionability (project, area, or resource).

• Distill: Find the essence of what a resource communicates.

• Express: Show your work based on the knowledge you gained.

Unless you’re a super memory genius, I strongly recommend looking into building a second brain. As we discussed, our brain should be for having ideas, not holding them.

Maybe the ideas in this post won’t fit you exactly, and you’ll come up with your own way of tracking notes. However you approach it, building a second brain has been incredibly important for me, and it might be for you as well.

📚 Resources

More From the Beyond the Code Category

• Don’t Forget About Your Mental Health

• Survivor Bias

• The XY Problem

Sources

• Building a Second Brain

Explore Further

• Create Your Own Second Brain

• Your Resource Guide to Building a Second Brain

❤️ If you enjoyed the post, please consider giving it a like. It’s a helpful signal to decide what to write next.

💬 Are you using a second brain system? If not, how do you keep track of your notes and ideas?

Leave a comment

Recently, I was going through the distributed systems reliability glossary brought by Antithesis & Jepsen, and I found their approach on availability models particularly interesting. Let’s dive into that.

Introduction

Here are two database whitepapers:

• Megastore, a Google storage system:

• Dynamo, an Amazon key-value store:

Both of these whitepapers use the term highly available. Of course, as a reader, we may expect these two whitepapers to mean the same thing when discussing high availability. After all, an orange is an orange, an apple is an apple, and a highly available system should be a highly available system. Yet, that’s not the case, and each paper means something different.

Let’s first look at what availability means, then discuss why high availability is a vague concept, and finally explore the different availability models.

Availability

In The CAP Theorem, we already discussed availability as:

Every request receives a non-error response, even if it may not contain the most up-to-date data.

However, this definition missed a crucial dimension: response time. Technically, a system could be available even if it responds after an hour. While such a system technically provides a non-error response, it fails to deliver a usable experience, severely compromising its practical availability from a user perspective.

This is where The PACELC Theorem offers a more practical perspective on availability. PACELC highlights that:

• In the presence of a partition, a system must choose between availability and consistency.

• In the absence of partition, a system must choose between latency (the upper-bound limit during which a request should receive a non-error response) and consistency.

So latency becomes part of availability. And that makes sense, right? If a system is too slow, it’s effectively down for the user. Availability isn’t just about uptime; it's also about whether the system is responsive in a meaningful timeframe.

High Availability

The term high availability is vague. Does it mean 99.9% uptime? 99.999%?

ScyllaDB, in their technical glossary says it’s about maintaining levels of uptime that exceed normal SLAs. That’s fine, but it’s also easy to game.

Say we define an SLA at 50%, then run at 80%. Technically, we exceeded it. Yet, does that mean we’re offering a highly available database? Probably not.

The Antithesis reliability glossary defines high availability as a system that is available more often than a single node. I quite like that one. If we take the availability of our best node and our system does worse than that, it’s not really highly available. Simple and practical.

Still, it’s not perfect. Let’s say we have five nodes, each available 50% of the time. With a write quorum of two, our write availability might still reach ~80%. But we’d still be down one request out of five. Hard to sell that as high availability.

To bring more clarity to the conversation, Antithesis introduced a set of availability models.

Availability Models

An availability model is something to help us define when an operation should succeed.

What do we mean by operation? It’s simply a request made to the system. That could be a read, a write, a ping, whatever the system is supposed to handle. Instead of thinking in terms of the whole system being up or down, we look at whether a specific request can succeed, even when parts of the system are failing.

Let’s explore three availability models.

Majority available

Definition: A system is majority available if when a majority of non-faulty nodes can communicate with one another, these nodes can execute some operations.

Consider a database composed of five nodes:

In the nominal case, everything works: a client connects, and the database can process all operations.

Now, imagine two nodes go down. Maybe they crash, maybe there’s a network issue causing a partition:

We’re left with three nodes that can still talk to each other. That’s a majority. If the database can still perform operations in this situation, we say it’s majority available.

That’s how the Megastore whitepaper defined highly available: being majority available.

This model is often used when consistency matters. For example, when a write or a leader election requires a majority to agree before responding to the client. So even if some nodes are unreachable, the system can still make safe progress as long as the majority is intact.

Total availability

Definition: A system is totally available if every non-faulty node can execute any operation.

Let’s take the same setup: a database with five nodes. This time, three of them are faulty:

In a majority available model, we can’t do much as we don’t have a majority. But in a totally available model, the system can still handle operations.

Indeed, in this model, each non-faulty node can act on its own. It doesn’t need to coordinate with others or wait for a network round-trip. This model favors latency. Just handle the request and move on.

That’s how the Dynamo whitepaper defined highly available: being totally available.

The tradeoff is consistency. Totally available systems can’t enforce strong guarantees because the nodes don’t necessarily sync before responding. That’s why this model typically goes hand in hand with weaker consistency models.

Sticky Available

Definition: A system is sticky available if whenever a client’s transactions are executed against a copy of database state that reflects all of the client’s prior operations, it eventually receives a response, even in the presence of indefinitely long partitions.

Let’s look at an example. Two clients, A and B, are connected to a database and make updates over time:

Sticky available means:

• After update 5, client A will eventually get a response that reflects at least updates 1, 4, and 5.

• After update 3, client B will eventually get a response that reflects at least updates 2 and 3.

How can we achieve that? It depends on how replication is handled by the system.

Fully Replicated System

In a fully replicated system, all nodes store the full dataset. Sticky availability can be achieved by making sure a client always talks to the same node.

Here’s the same example again, but now with client A always connected to node 1, and client B to node 2:

Here, node 2 hasn’t yet replicated update 5, and node 1 hasn’t yet replicated update 3. Still, since each client sticks to one node, they eventually see a consistent view of their own operations, despite possible failures such as a partition between the nodes.

Partially Replicated System

Now, let’s discuss a partially replicated system where nodes are replicas for subsets of data items.

Here’s a (dummy) partitioning system where even-numbered updates go to node 1 and odd-numbered ones to node 2:

Here, clients can’t just stick to a single node. Instead, they must maintain stickiness with a single logical copy of the database, which may consist of multiple nodes.

Clients can also help implement this model by acting as servers themselves. For example, a client could cache its own reads and writes, allowing it to return responses even during indefinitely long partitions.

Summary

• Highly available is too vague; watch out when you read or hear it. It might mean different things depending on the system or author.

• Majority available means a majority of nodes can still perform some operations. This model supports stronger consistency.

• Totally available means each non-faulty node can handle requests independently. It favors latency, but usually comes with weaker consistency.

• Sticky available means clients can make progress as long as they keep talking to a replica that reflects their own history.

• Availability models help us reason at the operation level, not just the system level. What matters is which operation can succeed, and under what conditions.

📚 Resources

More From the Distributed Systems Category

• The PACELC Theorem

• Exploring Database Isolation Levels

• Latency and User Experience

Sources

• A distributed systems reliability glossary - Antithesis

Explore Further

• High Availability Database Definition - ScyllaDB

• Consistency Models - Jepsen

❤️ If you enjoyed the post, please consider giving it a like. It’s a helpful signal to decide what to write next.

💬 When someone says their system is “highly available,” what do you assume they mean?

Leave a comment

Let me ask you a simple question: what makes a code beautiful?

Try thinking about this question (at least a little) before scrolling down.

You probably came up with a few characteristics: It’s readable and modular. Maybe it’s well tested. Or maybe it follows clean code principles or other familiar practices.

What if I told you that beautiful code is just… average code? That might sound silly, but let’s walk through it.

Attractive Faces Are Only Average is a foundational study by Judith H. Langlois and Lori A. Roggman in the psychology of facial attractiveness. They created composite images by averaging multiple individual faces—noses, eyes, cheekbones—all mathematically averaged.

These composites were consistently rated as more attractive than the individual faces used to make them. In fact, the more individual faces were averaged into a composite, the more attractive the resulting face became.

This suggests that facial features close to the population mean are generally preferred.

Why? The main theory is processing fluency: the easier something is for our brain to process, the more we like it. Average faces are easier to recognize and interpret, so they feel more pleasant.

This principle doesn’t stop at faces. It applies to advertising, art, or even decision-making. When something is easy to process, brain imaging studies show that it activates reward-related regions. We enjoy it more, without even realizing why.

So what happens when we apply this principle to code?

It means that the more average the code is, the more it aligns with what we’ve seen before, and the more beautiful it feels. The less surprising it is, the more pleasurable it is to read, understand, and maintain.

How can we write average code?

• Use standard naming conventions: Stick to familiar names so others instantly know what things are.

• Follow language idioms and common patterns: Write code the way others expect it to be written.

• Favor clarity over cleverness: The easier it is to read, the more enjoyable it is to work with.

• Be consistent: Predictability makes everything easier to understand.

• Minimize unnecessary novelty: The more our code looks like what we’ve seen before, the faster it gets processed and the more it’s liked.

Beautiful code is the kind that conforms to established patterns, idioms, and norms. It’s not flashy, novel, or overly clever. It’s clear, familiar, and quietly elegant. Just average.

What if next time, the best compliment your pull request could receive wasn’t that it was “brilliant” but simply “average”?

📚 Resources

More From the Software Engineering Category

• Cognitive Load: Reducing Mental Overhead in Software Design

• Readability: Understanding the What, the Why, and the How

• Premature Abstractions: Avoiding Unnecessary Complexity in Software Design

Sources

• Attractive Faces Are Only Average // The study mentioned in the post.

• Your Code as a Crime Scene // I read about this study from this book.

Explore Further

• Processing fluency

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 Does this idea resonate with you or does it feel wrong? Curious to hear your perspective.

Leave a comment

Last month, I was at the hospital with my partner for the birth of our newborn. During our stay, my partner experienced a specific symptom, and we wanted to understand what could be causing it. So we asked the midwife.

We immediately noticed the hesitation in her eyes. When she finally gave an answer, it came with a kind of forced confidence, and we both felt she wasn’t sure about it.

At our hospital, midwives do 12-hour shifts. So a few hours later, we asked the exact same question to the next midwife. Same hesitation, but this time, a different answer.

And so it went on. Every shift, we asked again. Every time, a different answer. Eventually, it even became a game between my partner and me: trying to guess what the next answer would be.

Until… The one. The one who broke this cycle.

We asked her the same question. She paused. Thought about it. And then said something unexpected:

It might seem counterintuitive, but these three words made us immediately trust her more than anyone else before1.

Twenty minutes later, she even came back to our room and said:

That brief exchange resonated with me.

In our field, we often put a lot of weight on posture. We build up our position as the go-to person for a codebase, a data model, or a framework. The more we know, the more we are seen as the one to consult or include in any related discussion.

But from that posture, admitting we don’t know something can feel like pulling out the bottom card in a house of cards. Suddenly, it feels like everything we built to earn that status might collapse.

Yet, if we take a step back, admitting we don’t know shouldn’t be seen as something shameful or embarrassing. In fact, it’s often the most responsible thing we can do.

• Pretending to know can lead to bad decisions. It creates false confidence and can steer a team in the wrong direction leading to possibly terrible outcomes.

• Authority isn’t built on knowing everything. It’s about being reliable and honest, someone others trust, and they can talk to with confidence.

• Teams work better when people feel safe to admit uncertainty. It fosters psychological safety among the team members, which Google identified as the top trait of effective teams.

• Curiosity + humility = real learning. Admitting we don’t know something is what keeps us learning and growing, no matter how experienced we are.

Whether it’s for us or others, next time we don’t know something, let’s be like that midwife: let’s just admit it. Without shame.

📚 Resources

More From the Beyond the Code Category

• Lateral Thinking

• Streetlight Effect

• 10 Rules I Learned About Technical Writing

Sources

• Understand team effectiveness

❤️ If you made it this far and enjoyed the post, please consider giving it a like.

💬 How comfortable are you with saying “I don’t know“?

Leave a comment