Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Over this series, you built a working LSM tree: you flush to persist the memtable to disk and compact to reclaim space.

Yet, you’ve been single-threaded so far. This week, we lift that constraint: flush and compaction will run in the background while you keep serving requests.

There are many ways to add concurrency. The approach here is to introduce a versioned, ref-counted catalog that lets readers take a stable snapshot while background flush/compaction proceeds.

A catalog holds references to:

• The current memtable.

• The current WAL.

• The current MANIFEST.

Each request pins one catalog version for the duration of the operation.

When a flush or compaction completes, the system creates a new catalog version. Old resources (e.g., obsolete SSTables) are not deleted immediately. Instead, each catalog tracks a refcount of in-flight requests. Once an old catalog’s refcount drops to zero and a newer catalog exists, you can safely garbage-collect the resources that appear in the old version but not in the new one.

For example, with two catalog versions (red = older version’s elements, blue = newer element”):

Once we can guarantee that catalog v1 is no longer referenced, we can delete the old MANIFEST, SST-2 and SST-3.

Another example: a flush produced a new memtable and WAL file:

In this case, once vatalog v1 has no remaining references, we can free the old memtable and delete the old WAL file.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Catalog

• Add a Catalog data structure that tracks:

  • Memtable reference.

  • WAL path.

  • MANIFEST path.

  • Version (monotonic).

  • Refcount of active readers.

• Implement a Catalog manager that keeps catalog versions in memory:

  • Acquire() → Catalog:

    • Pick the latest catalog.

    • Increment its refcount.

    • Return it.

  • Release(Catalog):

    • Decrement the refcount of the catalog.

    • If refcount is zero and there’s a new catalog version:

      • Remove the current catalog.

      • Remove elements present in the current catalog but not in the latest version (files, WAL, etc.)

  • NewCatalog(memtable, walPath, manifestPath):

    • Create a new catalog based on the provided data.

    • Assign a unique, monotonic version.

• At startup:

  • Read from the authoritative MANIFEST (latest MANIFEST file).

  • Treat any files not listed in MANIFEST as orphans and delete them.

  • Read all WAL files you still have on disk, in order, to rebuild the in-memory state.

  • Create the current catalog version from the reconstructed state.

  • Start the background worker.

Background Workers

In a nutshell, flush and compaction will move to the background. You’ll use internal queues plus worker pools to ensure no overlapping work on the same resources: at most one flush running at a time, and at most one compaction running at a time.

Compaction:

• Keep the same trigger: Every 10,000 update requests.

• Behavior:

  • Do not run compaction in the request path.

  • On compaction trigger: Post a notification to an internal queue and return.

• Worker:

  • A single background thread listens on the queue and runs the actual work.

  • Similar compaction process, except:

    • Do not overwrite the existing WAL file. Instead, create a new WAL-<version+1> file.

    • Create a new catalog that references the new WAL.

Flush:

• Keep the same trigger: When the memtable contains 2,000 entries.

• Behavior:

  • Do not run flush in the request path.

  • On flush trigger:

    • Allocate a new memtable and create a new WAL file for subsequent writes.

    • Post a notification to an internal queue.

    • Return immediately to the caller.

• Worker:

  • A single background thread listens on the queue and runs the actual work.

  • Similar flush process, except:

    • Do not overwrite the existing MANIFEST file. Instead, create a new MANIFEST-<version+1> file.

    • Create a new catalog referencing the new MANIFEST.

GET/PUT/DELETE

• Acquire a catalog from the manager.

• Do the operation using paths/refs from that catalog.

• Release the catalog.

Client & Validation

Concurrent requests make deterministic assertions harder. For example, suppose the validation file contains the following requests that can run in parallel:

What should you assert for GET foo: 1, 2, or 3?

To make validation deterministic, you will handle barriers: all requests before a barrier must finish before starting the next block. You will also relax GET checks: a GET is valid if it returns any value written for a key before the last barrier.

A similar example with barriers:

• The first two PUT requests run in parallel.

• The first barrier waits for both to complete.

• The first GET should accept either 1 or 2.

• The second GET request should accept only 3.

File Format

The new validation file is a sequence of blocks separated by BARRIER instructions:

(PUT|DELETE) <key> <value>
(PUT|DELETE) <key> <value>
...
BARRIER # New instruction
GET <key> [<value1> <value2> ... <valueK>]
GET <key> [<value1> <value2> ... <valueK>]
...
BARRIER

• All the lines between two barriers form a block.

• On BARRIER instruction, wait for all in-flight requests in the current block to finish before starting the next block.

• PUT/DELETE lines are issued in parallel within their block.

• GET lines are also issued in parallel within their block.

• GET <key> [<value1> <value2> ... <valueK means the response must be any one of the list values.

Download and run your client against a new file: concurrency.txt.

[Optional] Flush Anticipation

When the memtable reaches 80% of capacity:

• Pre-allocate the next memtable in memory.

• Pre-create/rotate to the next WAL on disk.

Wrap Up

That's it for the whole series.

You implemented a fully functional LSM tree:

• Started with a memtable (hashtable) and a flush that writes immutable SSTables to disk.

• Added a WAL for durability.

• Handled deletes and compaction to reclaim space.

• Introduced leveling and key-range partitioning to speed up reads.

• Switched to block-based SSTables with indexing.

• Added Bloom filters and replaced the memtable with a radix trie for faster lookups.

• Finally, introduced concurrency: a simple, single-threaded foreground path with flush and compaction running in the background.

I hope you had fun building it. Thank you for following the series, and special thanks to our partner, ScyllaDB!

Further Notes

To get more information on how things work in production databases, you can read how RocksDB keeps track of live SST files. The Catalog structure is inspired by RocksDB’s VersionSet.

Conflict resolution is one aspect we’re missing in the series (maybe as a follow-up?) A versioned catalog is enough for reads, but what about conflicting writes?

Suppose two clients, Alice and Bob, update the same key around the same time. A simple policy to resolve conflicts is latest wins. The database can serialize operations for the same key to ensure the latest request wins:

In this example, the database ends up with foo=baz as the latest state.

This approach works with one node. But what about databases composed of multiple nodes? Say the two requests go to two different nodes at roughly the same time:

With multiple nodes, the database must resolve conflicts consistently. There are two common ways:

1. Coordination via a leader (consensus): Route both writes to the same leader node, which solves the conflict and determines the end state.

2. Reconcile with comparable timestamps: Attach a timestamp to each write and store it with the key. By timestamp, we don’t mean relying on wall-clock time but a logical clock, so that “later“ is well-defined across nodes.

If we go with the second approach and start storing (key, value, timestamp) data, we also unlock something production systems use: consistent snapshots. A read can include a timestamp, and the database returns the last version at or before that time; hence, providing a consistent view of the data, even while flush/compaction runs in the background.

This pattern has a name: Multi-Version Concurrency Control (MVCC). It involves keeping multiple versions per key instead of only the last one, reading using a chosen point in time, and deleting old versions once they are no longer needed.

See how ScyllaDB handles timestamp conflict resolution for more information.

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Over the last few weeks, you refined your LSM tree to introduce leveling. In case of a key miss, the GET process requires the following steps:

• Lookup from the memtable.

• Lookup from all the L0 SSTables.

• Lookup from one L1 SSTable.

• Lookup from one L2 SSTable.

• Etc.

Last week, you optimized the lookups by introducing block-based SSTables and indexing, but a lookup is still not a “free” operation. Worst case, it requires fetching two pages (one for the index block and one for the data block) to find out that a key is missing in an SSTable.

This week, you will optimize searches by introducing a “tiny” level of caching per SSTable.

If you’re an avid reader of The Coder Cafe1, we already discussed a great candidate for such a cache:

• One that doesn’t consume too much memory to make sure we don’t increase space amplification drastically.

• One that is fast enough so that a lookup doesn’t introduce too much overhead, especially if we have to check a cache before making any lookup in an SSTable.

You will implement a cache using Bloom filters: a space-efficient, probabilistic data structure to check for set membership. A Bloom filter can return two possible answers:

• The element is definitely not in the set (no false negatives).

• The element may be in the set (false positives are possible).

In addition to optimizing SSTable lookups, you will also optimize your memtable.

In week 2, you implemented a memtable using a hashtable. Let’s get some perspective to understand the problems of using a hashtable:

• A memtable buffers writes.

• As it’s the main entry point for writes, a write has to be fast. → OK: a hashtable has average O(1) inserts, plus O(k) (k: the length of the key) for hashing.

• For reads, doing a key lookup has to be fast → OK: average O(1) lookups, plus O(k) to hash.

• Doing range scanning operations (week 5, optional work), such as: “Give me the list of keys between bar and foo“ → A hashtable, because it’s not an ordered data structure, is terrible: you end up touching everything so O(n) with n the number of elements in the hashtable.

• Flush to L0 → A hashtable isn’t ordered, so it requires sorting all the keys (O(n log n) with n the number of elements) to produce the SSTables.

Because of these negative points, could we find a better data structure? Yes! This week, you will switch the memtable to a radix trie (see Further Notes for a discussion on alternative data structures).

A trie is a tree-shaped data structure usually used to store strings efficiently. The common example to illustrate a trie is to store a dictionary. For example, suppose you want to store these two words:

• bake

• baker

Despite that baker starts with the same four letters, you need to store a total of 4 + 5 = 9 letters.

Tries optimize the storage required by sharing prefixes. Each node stores one letter. Here’s an example of a trie storing these two words in addition to the word foo (* nodes represent the end of a word):

As you can see, we didn’t duplicate the first four letters of bake to store baker. In this very example, instead of storing 9 letters for bake and baker, we stored only five letters.

Yet, you’re not going to implement a “basic” trie for your memtable; instead, you will implement a compressed trie called a radix trie (also known as a patricia2 trie).

Back to the previous example, storing one node (one square) has an overhead. It usually means at least one extra field to store the next element, usually a pointer.

In the previous example, we needed 11 nodes in total, but what if we could compress the number of nodes required? The idea is to combine nodes with a single child:

This new trie stores the exact same information, except it requires 6 nodes instead of 11. That’s what radix tries are about.

To summarize the benefits of switching a memtable from a hashtable to a radix trie:

• Ordered by design: Tries keep keys in order and make prefix/range lookups natural, which helps for SCAN and for streaming a sorted flush.

• No rebalancing/rehashing pauses: The shape doesn’t depend on insertion order, and operations don’t need rebalancing; you avoid periodic rehash work.

• Prefix compression: A radix trie can cut duplicated key bytes in the memtable, reducing in-memory space.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Bloom Filter

Let’s size the Bloom filter. You will target:

• p (false-positive rate) = 1%

• n (max elements per SSTable) = 1,953

• k (hash functions) = 5

Using the formula from the Bloom Filters post:

We get m ≈ 19,230 bits, i.e., 2,404 B. We will round up to 2,496 B (39 × 64 B), so the bitset is a whole number of cache lines.

NOTE: Using k=7 would shave only ~2–3% space for ~40% more hash work, so k=5 is a good trade-off.

To distribute elements across the bitvector, you will use the following approach. You will use xxHash64 with two different constant seeds to get two base hashes, then derive k indices by double hashing (pseudo-code):

const m: uint64 = 19230
const k: int    = 5

const seed1: uint64 = 0x9E3779B185EBCA87
const seed2: uint64 = 0xC2B2AE3D27D4EB4F

function bits(key: string): int[] {
    h1: uint64 = xxhash64(key, seed1)
    h2: uint64 = xxhash64(key, seed2)

    // Initialize an integer array of k elements.
    idx: int[] = new int[k]
    for i from 0 to k-1:
        idx[i] = (h1 + i*h2) % m
    return idx
}

The required changes to introduce Bloom filters:

• Startup:

  • For each SSTable in the MANIFEST, cache its related Bloom filter in memory.

    Since each Bloom filter requires only a small amount of space, this optimization has a minimal memory footprint. For example, caching 1,000 Bloom filters of the type you designed requires less than 2.5 MB of memory.

• SSTable creation:

  • For each new SSTable you write, initialize an empty bitvector of 2,496 B.

  • Build the Bloom filter in memory as you emit the keys (including tombstones):

    • Compute idx based on the key.

    • For each i, set bit at position idx[i].

  • When the SSTable is done, persist a sidecar file next to it (e.g., /l1/sst-3.sst and /l1/sst-3.bloom) and fsync the file.

  • Update the cache containing the Bloom filters.

• Compaction:

  • Delete from memory the Bloom filters corresponding to deleted SSTables.

• Lookup:

  • Before reading an SSTable:

    • Compute idx based on the key.

      • If all the bits of idx are set: The key may be present, therefore, proceed with your normal lookup in the SSTable.

      • Otherwise: Skip this SSTable.

Trie Memtable

Now, let’s replace your hashtable with a trie.

Node shape

• label: Compressed edge fragment.

• children: A map keyed by the next character after label to a node.

• state: An enum with the different possible values:

  • Empty: The node is just a prefix, no full key ends here.

  • Value: A full key exists at this node.

  • Tombstone: This key was explicitly deleted.

• value: If state is Value, the corresponding value.

Root is a sentinel node with an empty label.

Trie Operations

• update(key, val):

  • Walk from the root, matching the longest common prefix against label.

  • If partial match in the middle of an edge, split once: Create a parent with the common part, two children: the old suffix and the new suffix.

  • Descend via the next child (next unmatched character).

  • At the terminal node: set value=val and tombstone=false.

• get(key):

  • Walk edges by longest-prefix match. If an edge doesn’t match, return not found.

  • At the terminal node:

    • If state == HasValue: return value

    • If state == Empty or state == Tombstone, return not found.

• delete(key):

  • Walk as in get. If the path doesn’t fully exist, create the missing suffix nodes with state = Empty so that a terminal node exists.

  • At the terminal node: set state = Tombstone (you may have to clear value).

Flush process:

• In-order traversal:

  • HasValue: Emit (key, value).

  • Tombstone: Emit tombstone.

  • Empty: Emit nothing.

Client & Validation

There are no changes to the client. Run it against the same file (put-delete.txt) to validate that your changes are correct.

[Optional] Variable Bloom Filter Seed

Use per-SSTable random seeds for the Bloom hash functions. Persist them in the Bloom filter files.

[Optional] Blocked Bloom Filters

In Bloom Filters, you introduced blocked Bloom filters, a variant that optimizes spatial locality by:

• Dividing the bloom filter into contiguous blocks, each the size of a cache line.

• Restricting each query to a single block to ensure all bit lookups stay within the same cache line.

Switch to blocked Bloom filters and see the impacts on latency and throughput.

[Optional] SCAN

If you implemented the SCAN operation from week 5 (optional work), wire it to your memtable radix trie.

Wrap Up

That’s it for this week! You optimized lookups with per-SSTable Bloom filters and switched the memtable to a radix trie, an ordered data structure.

Since the beginning of the series, everything you built has been single-threaded, and flush/compaction remains stop-the-world. In two weeks, you will finally tackle the final boss of LSM trees: concurrency.

Further Notes

If you want to dive more into tries, Trie Memtables in Cassandra is a paper that explains why Cassandra moved from a skip list + B-tree memtable to a trie, and what it changed for topics such as GC and CPU locality.

A popular variant of radix trie is the Adaptive Radix Tree (ART): it dynamically resizes node types based on the number of children to stay compact and cache-friendly, while supporting fast in-memory lookups, inserts, and deletes. This paper (or this summary) explores the topic in depth.

You should also be aware that tries aren’t the only option for memtables, as other data structures exist. For example, RocksDB relies on a skip list. See this resource for more information.

About Bloom filters, some engines keep a Bloom filter not only per SSTable but per data-block range as well. This was the case for RocksDB’s older block-based filter format (source). RocksDB later shifted toward partitioned index/filters, which partition the index and full-file filter into smaller blocks with a top-level directory for on-demand loading. The official doc delves into the new approach.

Next: Week 8: Concurrency

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

In week 2, you used JSON as the SSTable format. That works for document databases, but the overhead of this serialization format doesn’t make it the best choice for your storage engine:

• Best case: You stream the file and linearly scan entries until you find the key, but a miss means scanning the entire file.

• Worst case: You read the whole file and parse everything, then search for the key.

This week, you will switch to block-based SSTables. Data will be chunked into fixed-size blocks designed to fit within a single disk page. The main benefits:

• Efficient I/O: Each lookup can fetch a complete block with a single page read.

• Predictable latency: Since every block maps to exactly one page, each read involves a fixed, bounded amount of I/O, improving latency consistency.

• Smaller on disk: Binary encoding typically compresses better than JSON.

• Integrity: Per-block checksums detect corruption without requiring a re-read of the file.

• Caching: Hot SSTable blocks are cached in a memory-based block cache to reduce I/O and decompression overhead.

Alongside the data blocks, you will maintain a small index that stores the first key of each block and its corresponding offset, allowing lookups to jump directly to the relevant block without scanning all of them.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Assumptions

• Fixed 64-byte keys and values: This alleviates a lot of logic to keep fixed-size blocks, making the implementation easier to write and reason about.

• Because of the week 1 assumption (keys are lowercase ASCII strings), each character is one byte, which also makes the implementation easier.

SSTable Format

A block-based SSTable will be composed of:

• One index block (first 4 KB page)

• Multiple data blocks (each 4 KB)

Each block has a fixed size of 4 KB. Aligning blocks to 4 KB means a disk read can fetch a block in one page. If blocks are not aligned, a read may span two pages.

Here’s the file layout at a glance:

Offset 0             4096                 8192                 ...
┌─────────────────────┬────────────────────┬────────────────────┬──────
│ Index block (4 KB)  │ Data block 0 (4 KB)│ Data block 1 (4 KB)│ ...
└─────────────────────┴────────────────────┴────────────────────┴──────

Index Block

The layout of an index block (4 KB):

• blockCount: The number of data blocks in the SSTable.

• A set of key entries (64 B), each being the first key of the corresponding data block. Entries are sorted by key and used to decide which block to fetch during a lookup.

To make the index fit into a single 4 KB page, it must contain at most 63 entries.

Here’s the layout (note this is a binary layout; newlines are used only for the representation):

<blockCount><pad>
|----1B----||63B|

# First index entry
<firstKey> # Right-padded with 0x00
|--64B--|

# Second index entry
<firstKey> # Right-padded with 0x00
|--64B--|

# Third index entry, etc.

# Padding with 0x00 to reach exactly 4096 bytes

NOTE: If you’re not familiar with the concept of padding: it’s filling unused bytes (here with 0x00) so fields and blocks have fixed sizes.

blockCount has a value between 0 and 63. If you encoded 63 as text, you would need two bytes (‘6’ = 0x36 and ‘3‘ = 0x33). Instead, you can store it as a binary integer so it fits in one byte: 0x3f.

Same layout, with explicit offsets:

Offset  Size  Field
0       1     blockCount
1       63    padding (0x00) to 64-byte alignment
64      64    Entry 0: firstKey of data block 0 (right-padded to 64B)
128     64    Entry 1: firstKey of data block 1 (right-padded to 64B)
192     64    Entry 2: firstKey of data block 2 (right-padded to 64B)
...     64    ...
64+64*n 64    Entry n: firstKey of data block n (right-padded to 64B)

# If fewer than 63 entries, fill remaining bytes with 0x00 up to 4096B.

An example of an SSTable with three data blocks, hence three entries. Remember: this is binary; newlines are for readability only:

3<63 0x00>                           # 1B key + 63B pad  = 64B slot
aaa<61 0x00>                         # 3B key + 61B pad  = 64B slot
everyLittleIslandSeemsEmpty<37 0x00> # 27B key + 37B pad = 64B slot
hello<59 0x00>                       # 5B key + 59B pad  = 64B slot
<3840 0x00>                          # Tail pad: 4096 - (4*64)

This index block indicates:

• Block 0 starts with the key aaa.

• Block 1 starts with the key everyLittleIslandSeemsEmpty.

• Block 2 starts with the key hello.

You don’t need to store per-block offsets. Because the index is stored on a 4 KB page and every data block is exactly 4 KB and written contiguously, offsets can be calculated this way (blockId starts at 0):

offset(blockId) = (blockId + 1) × 4096

Therefore:

• Block 0 starts at offset 4096.

• Block 1 starts at offset 8192.

• Block 2 starts at offset 12288.

Data Blocks

Now, let’s focus on data blocks.

In addition to the key-value entries, reserve 8 bytes in the block at the start to store a CRC computed over entryCount + all entries; this lets you verify data integrity on read.

The layout of a data block (4 KB per block):

• Header (128 B):

  • CRC-64 (8 B): A checksum computed over bytes [8..4096). You can choose any standard variant (e.g., CRC-64/ECMA-182).

  • entryCount (1 B): the number of entries in this block (0..31).

  • Padding (119 B).

• Entries area (31 x 128 B = 3968 B), each entry is:

  • key (64 B, right-padded).

  • value (64 B, right-padded).

The last data block may contain fewer than 31 entries (entryCount < 31), but always pad with zeros to reach exactly 4 KB. This guarantees one-page reads and prevents errors across read modes (e.g., SIGBUS with mmap).

The layout of a data block (again, newlines are used only for the representation):

<CRC64><entryCount><pad>
|--8B--|----1B----|119B|

<key><value> # Keys and values are right-padded with 0x00
|-64B|-64B|
<key><value> # Keys and values are right-padded with 0x00
|-64B|-64B|
<key><value> # Keys and values are right-padded with 0x00
|-64B|-64B|
# ...
# Zero-fill unused 128B entry slots so block size = 4096B

Same layout, with explicit offsets:

Offset  Size  Field
0       8     CRC64: CRC over [8..4096)
8       1     entryCount: 0..31 valid entries
9      119   padding

# Entries area (31 slots × 128B = 3968B)
128     64    Entry 0: key (right-padded to 64B)
192     64    Entry 0: value (right-padded to 64B)

256     64    Entry 1: key (right-padded to 64B)
320     64    Entry 1: value (right-padded to 64B)

384     64    Entry 2: key (right-padded to 64B)
448     64    Entry 2: value (right-padded to 64B)
...     ...   ...

# If entryCount < 31, zero-fill remaining 128B slots so block size = 4096B.

An example of a block composed of three key-value pairs:

0x42F0E1EBA9EA3693<entryCount=3><119 0x00>
alastor<57 0x00>foo<61 0x00>
aristide<56 0x00>bar<61 0x00>
armelle<57 0x00>z<63 0x00>
<3584 0x00> # (31 - 3) × 128B = 3584B

Note that because the index block holds at most 63 key entries, an SSTable can have at most 63 data blocks. With 31 entries per block, that caps an SSTable at 63 × 31 = 1,953 entries.

A tombstone is represented by a value of 64 bytes all set to 0x00. Due to this sentinel, the all-zero value is reserved and cannot be used as an application value from this week onward.

GET

Searching for a value doesn’t change (memtable → L0 → L1, etc.). What changes is how you read one SSTable (remember: from L1, you only need to read one SSTable per level because of non-overlapping key ranges).

The process to read from an SSTable:

1. Read blockCount.

2. Binary search the index in [0, blockCount-1] to find the largest firstKey ≤ key and get blockId.

   • If not found (e.g., first index key is bac and your key is aaa), return a miss for this SSTable.

3. Compute the block offset:
   blockOffset(blockId) = (blockId + 1) × 4096.

4. Fetch the corresponding 4 KB block.

5. Verify CRC before using the block:

   • Compute CRC64 over bytes [8..4096).

   • Compare with the 8-byte CRC stored at offset 0..7. If it doesn’t match, fail the read for this SSTable.

6. Read entryCount.

7. Binary search the entries in [0, entryCount-1] for the key.

8. Return the corresponding value or a miss.

Compaction

Last week, you split at 2,000 entries during the compaction process. This week, because a single SSTable is limited to 1,953 entries, change the split threshold to 1,953.

Client & Validation

There are no changes to the client. Run it against the same file (put-delete.txt) to validate that your changes are correct.

[Optional] Variable-Length Key/Values

Drop the 64-byte constraint: store a length-prefixed key and value per entry (short header with key length and value length).

Keep entries sorted and include the lengths in your checksum.

[Optional] No Sentiel Values for Tombstones

Tombstones are currently represented by a sentinel value (a 64-byte all-zero value), which prevents storing an actual empty value.

Instead, avoid reserving any value for deletes: add an explicit entry type per record (value or tombstone).

[Optional] Compression

Now that the format is binary, compression becomes more effective and saves more space.

As an optional task, compress each data block independently so lookups still touch only one block:

• Record each block’s offset and compressed size in the index.

• Read just those bytes, decompress, and search.

This packs more logical blocks into each cached page, raising cache hit rates, reducing pages touched during scans, and smoothing read latency.

Wrap Up

That’s it for this week! You implemented block-based SSTables and indexing, gaining benefits like more efficient I/O and reduced write amplification.

In two weeks, you will focus on improving read performance by adding a layer that can tell whether an SSTable is worth parsing, and say goodbye to your hashtable-based memtable, replacing it with a more efficient data structure.

Further Notes

For a production-grade implementation of block-based SSTables, see RocksDB’s block-based SSTable format. It details block layout, per-block compression, and how the index stores offsets and sizes.

You can also check out ScyllaDB’s SSTables v3 docs. ScyllaDB maintains a small in-memory summary of sampled keys to narrow the search, then uses the on-disk index to locate the exact block. This provides a nice contrast to our single-page index and illustrates how to scale when SSTables grow large.

For a deeper look at how things work in practice in terms of directory structure, you can explore the ScyllaDB SSTables directory structure, which shows how metadata and data are organized on disk.

Regarding CRC read failures, we mentioned that a checksum mismatch should simply cause the read to fail for that SSTable. In real systems, databases rely on replication to handle corruption. When multiple replicas exist, a system can recover by using data from an intact replica if one becomes corrupted or unavailable. Upon detecting a checksum mismatch, the system discards the corrupt replica and rebuilds it from a healthy one. This approach only works as long as a valid replica exists, which is why frequent checksum verification is critical: it ensures corruption is caught and repaired as early as possible, before it propagates.

Next: Week 7: Bloom Filters and Trie Memtable

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

I’ll also give a talk there, so feel free to join!

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Last week, you implemented deletion and compaction, making sure the LSM tree wouldn’t grow indefinitely.

Still, there’s a weak spot: in the worst-case scenario (e.g., on a key miss), a single read has to scan all SSTables. To address this, you will implement leveling, a core idea in LSM trees.

Instead of a single flat list of SSTables, leveling stores data across multiple levels: L0, L1, L2, etc.

• L0 gets compacted to L1 and makes space for future memtable flushes.

• L1 gets compacted to L2 and makes space for L0 compaction.

• L2 gets compacted to L3 and makes space for L1 compaction.

• …

• Ln-1 gets compacted to Ln and makes space for Ln-2 compaction.

This process is called level compaction.

Something important to understand: L0 is slightly different from all the other levels. L0 is created during memtable flushes. If a key already exists at L0 and also in the memtable, the next flush can write that key again to a new L0 file. In other words, L0 can have overlapping keys.

For all the other levels (L1 to Ln), that’s not the case. They are created by compaction, which removes duplicates and produces non-overlapping key ranges. In this week’s simplified design, an Li-1 to Li compaction takes all SSTables from Li-1 and Li, performs a k-way merge, then rewrites Li fully. As a result, each key appears at most once per level from L1 downward.

What’s the consequence of non-overlapping keys? You can improve lookups using a simple range-to-file mapping, for example:

• Keys from aaa to bac are stored in this SSTable.

• Keys from bac to cad are stored in this SSTable.

• etc.

With this setup, a read checks only one SSTable per level from L1 to Ln. L0 is the exception due to overlaps, so a read may still need to scan all L0 SSTables.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Leveling

• Limit the number of levels to two:

  • L0, which may contain overlapping keys.

  • L1, no overlapping keys.

• Create a folder for each level: /l0, and /l1.

• Keep one global MANIFEST file at the root.

Manifest Layout

You will create a MANIFEST layout for both L0 and L1:

• L0 remains a simple list of SSTables.

• L1 allows key-range partitioning.

For example:

[L0]
sst-11.json
sst-12.json
sst-13.json

[L1]
aaa-karate: sst-1.json
karate-leo: sst-2.json
leo-yaml: sst-3.json

This MANIFEST indicates:

• L0 is composed of three SSTables:

  • /l0/sst-11.json.

  • /l0/sst-12.json.

  • /l0/sst-13.json.

• L1:

  • Keys between aaa (included) and karate (excluded) live in /l1/sst-1.json.

  • Keys between karate (included) and leo (excluded) live in /l1/sst-2.json.

  • Keys between leo (included) and yaml (excluded) live in /l1/sst-3.json.

Compaction

The main goal of the compaction process is to compact both L0 and L1. At the end, you should merge all the data from L0 and L1 into L1. L0 will be left empty.

When L0 reaches five full SSTable files (2,000 entries each), run an L0 → L1 compaction:

• Open iterators on all L0 and L1 SSTables.

• Apply the k-way merge algorithm:

  • Comparator:

    • Primary: key.

    • Tie-break (equal key):

      1. Prefer L0 over L1.

      2. At L0, prefer the newest SSTable.

  • Version order: any record from L0 is newer than records from L1. Within L0, newer files win (same as week 4).

  • Keep at most one record per key (newest wins).

  • Tombstones: because L1 is the bottom level, drop a tombstone if no older value for that key remains in the merge result.

  • Create new L1 SSTables with at most 2,000 entries.

  • When naming new L1 files, make sure they are unique. For example, if /l1 contains sst-1.json and sst-2.json, the first SSTable file created should be sst-3.json.

• Publish atomically:

  1. fsync each new L1 file

  2. fsync the /l1 directory.

  3. Update the MANIFEST atomically.

  4. fsync the MANIFEST file.

  5. fsync the root directory (the directory containing the MANIFEST file and /l0 and /l1 folders).

• Clean up:

  1. Delete obsolete L1 files, then fsync /l1.

  2. Delete all files in /l0, then fsync /l0.

Flush (memtable to L0)

The logic is unchanged from previous weeks. The only difference is that flush writes to L0 and updates the MANIFEST file in the [L0] section.

GET

• Check the memtable.

• If not found, scan all L0 files newest to oldest using section [L0] of the MANIFEST.

• If not found at L0:

  1. Use the section [L1] of the MANIFEST to choose the one shard that contains the key’s range, then read only that L1 file.

  2. Return the value if found; otherwise, return 404 Not Found.

Client & Validation

There are no changes to the client. Run it against the same file (put-delete.txt) to validate that your changes are correct.

[Optional] Configurable Number of Levels

Introducing leveling has a fundamental impact on deletions. With a single level, compaction sees all versions of every key at once, so a tombstone can be dropped as soon as it has “killed“ every older record for that key. Yet, the rule we mentioned last week holds true: a tombstone can be evicted only after all data it shadows no longer exist on disk.

With multiple levels, compaction must propagate tombstones downward. It’s only at the bottommost level Ln that tombstones can be dropped, because only there you can prove they no longer shadow any other records.

As an optional task, make the number of levels configurable: L0, L1, …, Ln:

• Define a size ratio so each level has a target size larger than the previous one.

• Keep one directory per level: /l0, /l1, …, /ln.

• Keep a single global MANIFEST.

• When a level reaches its max number of SSTables (derived from the size ratio), compact that level into the next.

• Only drop tombstones at the bottommost level Ln. At any intermediate level Li with 0 ≤ i < n, propagate the tombstone downward during compaction.

[Optional] SCAN

Implement GET /scan?start={start}&end={end}:

• Return all keys between start (included) and end (excluded).

• Use put-delete-scan.txt to validate that your changes are correct. It introduces the SCAN keyword. For example:

  SCAN a-c aaa,bbb,bdx

  This line means: between a (included) and c (excluded), the keys are aaa, bbb, bdx (the output will always be sorted)

NOTE: If this route conflicts with GET /{key}, rename the single-key route to GET /keys/{key}.

Wrap Up

That’s it for this week! Your LSM tree is taking shape. You implemented leveling, a key LSM design idea, and refined compaction so reads are tighter and storage stays under control.

In two weeks, you will revisit the week 2 choice of JSON for SSTables. You will switch to block-based SSTables to reduce parsing and I/O overhead and add indexing within each SSTable.

Further Notes

We mentioned that, because of key overlaps, a read may still need to scan all L0 SSTables (e.g., key miss). This is the main reason why L0 is typically kept small. In general, each level is larger than the one above it by a fixed size ratio (e.g., 10×). Some databases even use less static mechanisms. For instance, RocksDB relies on Dynamic Leveled Compaction, where the size of each level is automatically adjusted based on the size of the oldest (last) level, eliminating the need to define each level’s size statically.

Regarding compaction, you should know that in real-world databases, it isn’t done in batch mode across all data. Let’s understand why.

Suppose you have four levels and a layout like this for one key:

• The key exists at L3.

• The key doesn’t exist at L2.

• The key is updated at L1.

• A tombstone is placed at L0.

You can’t compact L0 with L1/L2/L3 in one shot; that would mean checking every SSTable against every level.

What happens in reality is that compaction is a promotion process. In our example, the tombstone at L0 is promoted to L1. Implementations ensure that it either (a) is compacted together with the L1 SSTable it shadows, or (b) waits until that L1 data is promoted to L2. The same rule repeats level by level, until the tombstone reaches L3 and finally removes the shadowed value.

Meanwhile, it’s essential to understand that compaction is crucial in LSM trees. Let’s take some perspective to understand the reason. An LSM tree buffers writes in a memtable and flushes to L0. Compaction merges SSTables across levels to control read amplification. If compaction falls behind, L0 files accumulate, flushes slow down (or stall at file-count thresholds), write latency climbs, and in the worst case, you can observe write pauses. Not because the memtable is “locked,” but because the engine can’t safely create more L0 files until compaction catches up.

This is one of the reasons why the RUM conjecture we introduced last week is important.

• If you compact too eagerly, you burn a lot of disk I/O and lose the LSM’s write advantage.

• If you compact too lazily, you incur a penalty on your read path.

• If you compact everything all the time, you incur a space-amplification penalty during compaction roughly equal to the working set size.

Because compaction is so important, most key-value stores support parallel compactions across levels (except L0 → L1, which isn’t parallelized due to overlapping key ranges in L0).

You should also be aware that ongoing research keeps improving compaction. For example, the SILK: Preventing Latency Spikes in LSM Key-Value Stores paper analyzes why LSM systems can exhibit high tail latency. The main reason is that limited I/O bandwidth causes interference between client writes, flushes, and compactions. The key takeaway is that not all internal operations are equal. The paper explores solutions such as

• Bandwidth awareness: Monitor client I/O and allocate the leftover to internal work dynamically instead of static configuration.

• Prioritization: Give priority to operations near the top of the tree (flushes and L0 → L1 compaction). Slowdowns there create backpressure that impacts tail latency more than work at deeper levels.

Last but not least, what you implemented this week is called level compaction. Other strategies like tiered compaction exist, which merge SSTables based on their size and count rather than fixed levels. You can explore this great resource from Mark Callaghan, which dives deeper into the design trade-offs and performance characteristics of different compaction strategies in LSM trees.

Next: Week 6: Block-Based SSTables and Indexing

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Over the past few weeks, you built an LSM tree and three main components: a memtable, SSTables, and a WAL that records the same operations you keep in the memtable.

To prevent on-disk data from growing forever, you will implement compaction, a critical process in LSM trees. Compaction periodically merges SSTables to reclaim space and keep read performance predictable. For example, if key 1234 exists in every SSTable on disk:

Compaction drops duplicates and keeps only the newest record: 1234=baz.

In addition, you will implement a DELETE endpoint. Handling deletes in an LSM tree isn’t straightforward at all: SSTables are immutable. To preserve the append-only nature of LSM trees, deletions are written as tombstones: markers indicating a key was logically deleted. You write it to the WAL, keep it in the memtable, and propagate it during flush.

How should compaction work in the presence of tombstones? Suppose you have the following SSTables on disk: the key exists in SST-1, doesn’t exist in SST-2, exists in SST-3, and is deleted at SST-4:

If the key doesn’t exist in the memtable, the current state for 1234 is deleted. Now, imagine that during compaction, you merge SST-3 and SST-4. As the key is marked as deleted in the newest SSTable, you may decide to drop the tombstone, as it hides the key in SST-3:

Now, GET 1234 would do:

• Key doesn’t exist in the memtable → Continue.

• Key doesn’t exist in SST-5 → Continue.

• Key doesn’t exist in SST-2 → Continue.

• Key exists in SST-1 → Return foo (instead of 404 Not Found).

The fundamental rule is the following: during compaction, a tombstone can be evicted only after all data it shadows no longer exist on disk. Otherwise, dropping a tombstone too early can make an old value reappear. This is known as data resurrection: a key that “comes back to life” after a deletion.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Assumptions

• Flush and compaction should be single-threaded and stop-the-world operations: do not serve client requests until the operations complete.

DELETE Endpoint

Add DELETE /{key}:

1. Append a tombstone to the WAL file, with op=delete:

{”op”:”delete”,”key”:”k”}

2. fsync the WAL.

3. Update the memtable: Do not remove the key directly; mark it deleted with a tombstone.

4. Acknowledge the request.

Flush

During flush, carry tombstones into the new SSTable using a new deleted field. For example:

[
  ...
  { “key”: “foo”, “deleted”: true }, 
  ...
]

The keys must remain sorted.

Compaction

The goals of the compaction process for this week are the following:

• For each key, keep only the newest record.

• Drop records hidden by newer versions. This is where merging happens: the newest record wins, and older versions are evicted.

• Drop tombstones when no older value remains.

The compaction trigger is: every 10,000 update requests (PUT and DELETE, not GET), compact all SSTables.

Algorithm (k-way merge using a min-heap on key):

1. Open an iterator for each SSTable file known by the MANIFEST.

2. Push each iterator’s current record into a min-heap with the following comparator:

   • Primary: key.

   • Tie-break (equal key): Newest SSTable first based on MANIFEST order (to make sure an old value doesn’t win).

3. While the heap is not empty:

   1. Pop the smallest key k (this first pop is the newest version of k due to the tie-break).

   2. Drain all other heap entries whose key is k and discard them (older values).

   3. For the record you picked:

      • If it’s a tombstone, emit nothing for k.

      • Otherwise, emit the value for k.

   4. Advance only the iterators you drained for k and push their next records into the heap.

4. Stream emitted records (sorted) into new SSTables. Remember: the max entries in an SSTable should remain 2,000.

5. fsync each new SSTable file, then fsync its parent directory.

6. Update the MANIFEST atomically (see week 3).

7. Remove the old SSTable files.

GET

• Check the memtable:

  • If the key is marked as deleted, return 404 Not Found.

  • Else, return the value.

• Scan SSTables from newest to oldest, given the MANIFEST order (same as before).

• For the first record with the requested key:

  • If deleted=true, return 404 Not Found.

  • Else, return the value.

• If the key isn’t found, return 404 Not Found.

Startup

• When replaying the WAL, make sure to take into account tombstone values (op=delete).

Client & Validation

• Update your client to handle DELETE k lines → Send a DELETE request to /k.

• Download and run your client against a new file containing DELETE requests: put-delete.txt.

  NOTE: Refer to week 1 if you need to generate your own file with the number of lines you want.

Wrap Up

That’s it for this week! Your storage engine now supports deletes and a compaction mechanism that prevents unbounded growth.

The Coder Cafe will take a break for two weeks. On January 7th, you will continue exploring LSM trees and cover leveling. In your current implementation, a miss still scans all SSTables; therefore, you will also add key range partitioning to limit the number of SSTables that need to be checked during a lookup.

See you next year!

Further Notes

The compaction trigger you used was simple: every 10,000 PUT or DELETE requests. In real systems, compaction is usually driven by factors such as too many SSTable files, space pressure, or high read amplification.

Also, many systems add safeguards to keep compaction controlled and resource-efficient. For example, a common one is bounded fan-in (merging only a small, fixed number of SSTables per batch), so the engine never opens every file at once. Others track each SSTable’s first and last key to select only overlapping candidates, hence avoiding unrelated files.

Taking a step back, it’s interesting to note that the core LSM idea—append-only writes with regular compaction—shows up in many systems, even outside pure LSM trees. For example:

• Lucene: Immutable segments are created and later merged in the background, an LSM-like pattern, even though it isn’t an LSM tree per se.

• Memcached Extstore: Flushes values to free RAM, but keeps the hashtable, keys, and storage pointers in memory. It later compacts the data.

• Kafka: Rewrites segments to keep the latest value per key and drop older versions, which is conceptually similar to SSTable compaction.

Also, we briefly introduced the concept of key resurrection in the introduction. You should be aware that this is a common challenge with LSM trees. In real-world conditions, crashes, slow WAL truncation, and complex compaction can allow an old value to be replayed during recovery after its tombstone has been removed, leading to key resurrection. Here are two great references that delve more into this kind of problem:

• Preventing Data Resurrection with Repair Based Tombstone Garbage Collection

• Repair Time Requirements to Prevent Data Resurrection in Cassandra & Scylla

Another excellent reference is Acheron: Persisting Tombstones in LSM Engines. It shows how standard LSM compaction can leave tombstones stuck for long periods, so “deleted" data may still linger in lower levels and complicate compliance requirements such as GDPR/CCPA compliance. The paper introduces delete-aware techniques that prioritize pushing tombstones down the tree to make deletions persist more predictably.

Lastly, you can explore the RUM conjecture. Structurally, it’s similar to the CAP theorem: “three things, pick two”. In short, you can make a database excel at two of: reads, updates (insert/change/delete), and memory/space, but not all three at once. Make any two really good and the third gets worse; that’s an unavoidable trade-off. This helps explain why, for example, LSM trees optimized for fast updates and good space efficiency pay a cost in read performance due to read amplification.

That trade-off shows up in the design of the compaction process you implemented this week: you trade space and significant I/O for simplicity by compacting everything in one shot. This is fine for the example, but with 500GB of SSTables, you may need roughly another 500GB of free space during the merge in the worst case.

Next: Week 5: Leveling and Key-Range Partitioning

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Last week, you built the first version of an LSM: an in-memory memtable for recent writes, immutable SSTables on disk, and a MANIFEST file listing the SSTable files. However, if the database crashes, data in the memtable would be lost.

This week, you will focus on durability by introducing Write-Ahead Logging (WAL). A WAL is an append-only file on disk that records the same operations you keep in memory. How it works:

• On write, record it in the WAL and the memtable.

• On restart, you read the WAL from start to end and apply each record to the memtable.

Introducing a WAL is not free, though. Writes are slower because each write also goes to the WAL. It also increases write amplification, the ratio of data written to data requested by a client.

Another important aspect of durability is when to synchronize a file’s state with the storage device. When you write to a file, it may appear as saved, but the bytes may sit in memory caches rather than on the physical disk. These caches are managed by the OS’s filesystem, an abstraction over the disk. If the machine crashes before the data is flushed, you can lose data.

To force the data to stable storage, you need to call a sync primitive. The simple, portable choice is to call fsync, a system call that flushes a file’s buffered data and required metadata to disk.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

PUT

For the WAL data format, you won’t use JSON like the SSTables, but NDJSON (Newline-Delimited JSON). It is a true append-only format with one JSON object per line.

1. Append a record to the WAL file wal.db, opened with O_APPEND. Set the op field to “put”, and the key and value fields to the provided key and value. For example, writing k/v2:

   {”op”:”put”,”key”:”k”,”value”:”v1”}
   {”op”:”put”,”key”:”k”,”value”:”v2”} <- Latest write

2. fsync wal.db.

3. Update the memtable with the same logic as before:

   • If the key exists, update the value.

   • Otherwise, create a new entry.

4. Acknowledge the HTTP request.

Startup

• Create an empty wal.db file if it doesn’t exist.

• Replay the WAL from start to end. For each valid line, apply it to the memtable.

Flush

Keep the same flush trigger (2,000 entries) and the same logic (stop-the-world operation) as last week:

1. Write the new SSTable:

   1. Flush the memtable as a new immutable JSON SSTable file with keys sorted (same as before).

   2. fsync the SSTable file.

   3. fsync the parent directory of the SSTable to make the new filename persistent.

2. Update the MANIFEST atomically:

   1. Read the current MANIFEST lines into memory and append the new SSTable filename.

   2. Open MANIFEST.tmp with O_CREAT | O_TRUNC | O_WRONLY.

   3. Write the entire list to MANIFEST.tmp from the start.

   4. fsync MANIFEST.tmp.

   5. Rename MANIFEST.tmp → MANIFEST.

   6. fsync MANIFEST.

   7. fsync the parent directory of the MANIFEST.

3. Reset the WAL:

   1. Truncate the WAL to zero length.

   2. fsync the WAL file.

Client & Validation

If the server is unavailable, do not fail. Retry indefinitely with a short delay (or exponential backoff).

To assess durability:

• Run the client against the same input file (put.txt).

• Stop and restart your database randomly during the run.

• Your client should confirm that no acknowledged writes were lost after recovery.

[Optional] Partial WAL Records

Add a per-record checksum to each WAL record. On startup, verify records and stop at the first invalid/truncated one, discarding the tail.

For reference, ScyllaDB checksums segments using CRC32; see its commitlog segment file format for inspiration.

[Optional] Dangling Files

Regarding the flush process, if the database crashes after step 1 (write the new SSTable) and before step 2 (update the MANIFEST atomically), you may end up with a dangling SSTable file on disk.

Add a startup routine to delete any file that exists on disk but is not listed in the MANIFEST. This keeps the data directory aligned with the MANIFEST after a crash.

Wrap Up

That’s it for this week! Your storage engine is now durable. On restart, data that was in the memtable is recovered from the WAL. This is made possible by fsync and the atomic update of the MANIFEST.

Deletion is not handled yet. In the worst case, a miss can read all SSTables, which quickly becomes highly inefficient.

In two weeks, you will add a DELETE endpoint and learn how SSTables are compacted so the engine can reclaim space and keep reads efficient.

Further Notes

In your implementation, you used fsync as a simple “make it durable now“ button. In practice, fsync offers finer control both over what you sync and when you sync.

• What: fdatasync (or opening the file with O_DSYNC) persists the data without pushing unrelated metadata, which is usually what you want for WAL appends. You can go further with O_DIRECT + O_DSYNC to bypass the page cache and sync only the data you wrote, but that comes with extra complexity.

• When: While calling a sync primitive after every request is offered by systems that promise durability, it is often not the default. Many databases use group commit, which batches several writes into one fsync call to amortize the cost while still providing strong guarantees. For additional information, see A write-ahead log is not a universal part of durability by .

  For example, RocksDB provides options for tuning WAL behavior to meet the needs of different applications:

  • Synchronous WAL writes (what you implemented this week)

  • Group commit.

  • No WAL writes at all.

If you want, you can also explore group commit in your implementation and its impact on durability and latency/throughput, since this series will not cover it later.

Also, you should know that since a WAL adds I/O to the write path, storage engines use a few practical tricks to keep it fast and predictable. A common one is to preallocate fixed-size WAL segments at startup to:

• Avoid the penalty of dynamic allocation.

• Prevent write fragmentation.

• Align buffers for O_DIRECT (an open(2) flag for direct I/O that bypasses the OS page cache).

Next: Week 4: Deletes, Tombstones, and Compaction

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Before delving into this week’s tasks, it’s important to understand what you will implement. This week, you will implement a basic log-structured merge-tree (LSM tree).

At its core, an LSM tree is a data structure that prioritizes write efficiency by trading off some read complexity. It buffers writes in memory and uses append-only files on disk, then rewrites data during compaction. It consists of two main components:

• A mutable in-memory data structure called a memtable, used to store recent writes.

• A set of immutable SSTables (Sorted String Table) stored on disk.

Regularly, the current memtable is snapshotted, its entries are sorted by key, and a new immutable SSTable file is written.

In addition, a MANIFEST file is an append-only list of SSTable filenames. It tells the engine which SSTable files exist and in which order to read them, newest to oldest.

Why LSM trees shine for write-heavy workloads:

• Fast writes with sequential I/O: New updates are buffered in memory (memtable) and later written sequentially to disk during a flush (SSTable), which is faster than the random I/O patterns common with B-trees, for example.

• Decouples writes from read optimization: Writes complete against the memtable, while compaction work runs later (you will tackle that in a future week).

• Space and long-term efficiency: Compaction processes remove dead data and merge many small files into larger sorted files, which keeps space usage in check and sustains read performance over time.

For the memtable, you will start with a hashtable. In a future week, you will learn why a hashtable is not the most efficient data structure for an LSM tree, but it is a simple starting point.

For the SSTables, you will use JSON as the data format. Get comfortable with a JSON parser if you are not already.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Assumptions

• This week’s implementation is single-threaded. You will revisit that assumption later.

Memtable

Implement a hashtable to store PUT requests (create or update). You can probably reuse a lot of code from Week 1.

Flush

When your memtable contains 2,000 entries:

• Flush the memtable as a new immutable JSON SSTable file with keys sorted. The SSTable file is a JSON array of objects, each with two fields, key and value. Keys are unique within a file.

• For example, if your memtable contains the following entries:

  a = hello
  b = world
  z = 42

  You need to create the following SSTable:

  [
    { “key”: “a”, “value”: “hello” },
    { “key”: “b”, “value”: “world” },
    { “key”: “z”, “value”: “42” }
  ]

• Use a counter for the filename prefix, for example sst-1.json, sst-2.json, sst-3.json.

• After writing the new SSTable, append its filename to the MANIFEST (append only), then clear the memtable:

  sst-1.json
  sst-2.json
  sst-3.json <- new

For now, the flush is a stop-the-world operation. While the file is being written, do not serve reads or writes. You will revisit that later.

Startup

• Create an empty MANIFEST file if it doesn’t exist.

• Derive the next SSTable ID from the MANIFEST so you don't reuse the same filename.

GET

Update GET /{key}:

• Check the memtable:

  • If found, return the corresponding value.

  • If not found, read the MANIFEST to list SSTable filenames:

    • Scan SSTables from newest to oldest (for example sst-3.json, then sst-2.json, then sst-1.json). Use a simple linear scan inside each file for now. Stop at the first hit and return the corresponding value.

    • If still not found, return 404 Not Found.

Client & Validation

There are no changes to the client you built in week 1. Run it against the same file (put.txt) to validate that your changes are correct.

[Optional] Negative Cache

Keep a small LRU cache of known-absent keys (negative cache) between the memtable and SSTables. This avoids repeated disk scans for hot misses: after the first miss, subsequent lookups are O(1).

Implementation details are up to you.

[Optional] MANIFEST Cache

Instead of parsing the MANIFEST file for each GET request, you can cache the content in-memory.

Wrap Up

That’s it for this week! You have built the first version of an LSM tree: a memtable in memory, SSTable files written by regular flushes, and a MANIFEST that lists those SSTables.

For now, durability isn’t guaranteed. Data already flushed to SSTables will be read after a restart, but anything still in the memtable during a crash is lost.

In two weeks, you will make sure that any request acknowledged to a client remains in your storage engine, even after a restart.

Further Notes

The flush trigger you used was pretty simple: once the memtable contains 2,000 entries. In real systems, flushes can be triggered by various factors, for example:

• Some databases flush when the memtable reaches a target size in bytes, ensuring predictable memory usage.

• A flush can also occur after a period of time has passed. This occurs because the database eventually needs to release commit log segments. For tables with very low write activity, this can sometimes lead to data resurrection scenarios. Here’s an old issue from the ScyllaDB codebase that illustrates this behavior.

Regarding the model, this series assumes a simple key–value one: every PUT stores the whole value, so a GET just finds the newest entry and returns it. If you need a richer model (e.g., rows with many fields or collections), writes are often partial (patches) rather than full replacements. Therefore, reads must reconstruct the result by scanning newest to oldest and merging changes until all required fields are found or a full-write record is encountered.

Last but not least, in this series, you implicitly rely on client-side ordering: the validation client issues requests sequentially. Production KV databases typically attach a sequence number or a logical timestamp to each write to handle out-of-order arrivals, merging, and reconciling results. Pure wall-clock timestamps are convenient but brittle; see Kyle Kingsbury’s notes on clock pitfalls for a deeper dive.

Next: Week 3: Durability with Write-Ahead Logging

Sources

• The Log-Structured Merge-Tree (LSM-Tree) // The original LSM tree whitepaper.

• Log Structured Merge Tree - ScyllaDB // LSM tree definition from ScyllaDB technical glossary.

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Welcome to week 1 of Build Your Own Key-Value Storage Engine!

Let’s start by making sure what you’re about to build in this series makes complete sense: what’s a storage engine?

A storage engine is the part of a database that actually stores, indexes, and retrieves data, whether on disk or in memory. Think of the database as the restaurant, and the storage engine as the kitchen that decides how food is prepared and stored.

Some databases let you choose the storage engine. For example, MySQL uses InnoDB by default (based on B+-trees). Through plugins, you can switch to RocksDB, which is based on LSM trees.

This week, you will build an in-memory storage engine and the first version of the validation client that you will reuse throughout the series.

Your Tasks

💬 If you want to share your progress, discuss solutions, or collaborate with other coders, join the community Discord server (#kv-store-engine channel):

Join the Discord

Assumptions

• Keys are lowercase ASCII strings.

• Values are ASCII strings.

NOTE: Assumptions persist for the rest of the series unless explicitly discarded.

REST Endpoints to Implement

• PUT /{key}:

  • The request body contains the value.

  • If the key exists, update its value and return success.

  • If the key doesn’t exist, create it and return success.

  • Keep all data in memory.

• GET /{key}:

  • If the key exists, return 200 OK with the value in the body.

  • If the key does not exist, return 404 Not Found.

Client & Validation

Implement a client to validate your server:

• Read the testing scenario from this file: put.txt.

• Run an HTTP request for each line:

  • PUT k v → Send a PUT to /k with body v.

  • GET k v → Send a GET to /k. Confirm that v is returned. If not, something is wrong with your implementation.

  • GET k NOT_FOUND → Send a GET to /k. Confirm that 404 Not Found is returned. If not, something is wrong with your implementation.

• Each request must be executed sequentially, one line at a time; otherwise, out-of-order responses may fail the client’s assertions.

Input File Generation

If you want to generate an input file with a different number of lines, you can use this Go generator:

go run gen.go <format> <lines>

• <format> is the format to generate.

• <lines> is the number of lines.

At this stage, you need a put-type file, so for example, if you need one million lines:

go run gen.go put 1000000

[Optional] Metrics

Add basic metrics for latency:

• Record start and end time for each request.

• Keep a small histogram of latencies in milliseconds.

• At the end, print p50, p95, and p99.

This work is optional as there is no latency target in this series. However, it can be an interesting point of comparison across weeks to see how your changes affect latency.

Wrap Up

That’s it for this week! You have built a simple storage engine that keeps everything in memory.

In two weeks, we will level up. You will delve into a data structure widely used in key-value databases: LSM trees.

Next: Week 2: LSM Tree Foundations

❤️ If you enjoyed this post, please hit the like button.

Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.

Agenda

• Week 0: Introduction

• Week 1: In-Memory Store

• Week 2: LSM Tree Foundations

• Week 3: Durability with Write-Ahead Logging

• Week 4: Deletes, Tombstones, and Compaction

• Week 5: Leveling and Key-Range Partitioning

• Week 6: Block-Based SSTables and Indexing

• Week 7: Bloom Filters and Trie Memtable

• Week 8: Concurrency

Introduction

Welcome to The Coding Corner! This is our new section at The Coder Cafe, where we build real-world systems together, one step at a time.

Next week, we will launch the first post series: Build Your Own Key-Value Storage Engine.

Are you interested in understanding how key-value databases work? Tackling challenges like durability, partitioning, and compaction? Exploring data structures like LSM trees, Bloom filters, and tries? Then this series is for you.

Build Your Own Key-Value Storage Engine focuses on the storage engine itself; we will stay single-node. Topics such as replication and consensus are out of scope. Yet, if this format works, we may cover them in a future series.

The structure of each post will be as follows:

1. Introduction: The theory for what you are about to build that week.

2. Your tasks: A list of tasks to complete the week’s challenges. Note that you can complete the series in any programming language you want.

3. Further notes: Additional perspective on how things work in real systems.

Last but not least, I’m delighted to share that this series was written in collaboration with ScyllaDB. They reviewed the content for accuracy and shared practical context from real systems, providing a clearer view of how production databases behave and the problems they solve. Huge thanks to , Felipe Cardeneti Mendes, and ScyllaDB.

By the way, they host a free virtual conference called Monster Scale Summit, and the content is always excellent. If you care about scaling challenges, it’s absolutely worth registering! Also, if you’re interested in giving a talk, the CFP closes in two days.

On a personal note, this has been the most time-consuming project I have done for The Coder Cafe. I really hope you will enjoy it!

See you this Friday for a special post for Halloween and next Wednesday for the first post of the series.

Next: Week 1: In-Memory Store

❤️ If you enjoyed this post, please hit the like button.