Build Your Own Key-Value Storage Engine
Eight Weeks to a Working Key-Value Storage Engine
Welcome to The Coding Corner! This is our new section at The Coder Cafe, where we build real-world systems together, one step at a time.
Next week, we will launch the first post series: Build Your Own Key-Value Storage Engine.
Are you interested in understanding how key-value databases work? Tackling challenges like durability, partitioning, and compaction? Exploring data structures like LSM trees, Bloom filters, and tries? Then this series is for you.
Build Your Own Key-Value Storage Engine focuses on the storage engine itself; we will stay single-node. Topics such as replication and consensus are out of scope. Yet, if this format works, we may cover them in a future series.
The structure of each post will be as follows:
Introduction: The theory for what you are about to build that week.
Your tasks: A list of tasks to complete the week’s challenges. Note that you can complete the series in any programming language you want.
Further notes: Additional perspective on how things work in real systems.
Last but not least, I’m delighted to share that this series was written in collaboration with ScyllaDB. They reviewed the content for accuracy and shared practical context from real systems, providing a clearer view of how production databases behave and the problems they solve. Huge thanks to Cynthia Dunlop, Felipe Cardeneti Mendes, and ScyllaDB.
By the way, they host a free virtual conference called Monster Scale Summit, and the content is always excellent. If you care about scaling challenges, it’s absolutely worth registering! Also, if you’re interested in giving a talk, the CFP closes in two days.
Monster Scale Summit
Curious how leading engineers tackle extreme scale challenges with data-intensive applications? Join Monster Scale Summit (free + virtual). It’s hosted by ScyllaDB, the monstrously fast and scalable database.
On a personal note, this has been the most time-consuming project I have done for The Coder Cafe. I really hope you will enjoy it!
See you this Friday for a special post for Halloween and next Wednesday for the first post of the series.
Agenda
Week 5: Leveling and Key-Range Partitioning
Week 6: Block-Based SSTables and Indexing
Week 7: Bloom Filters and Trie Memtable
Week 8: Concurrency
❤️ If you enjoyed this post, please hit the like button.
Hi Teiva, what amazing news! It really aligns with I have been studying. I am reading Designing Data-Intensive Applications and learning more about Fundamentals of Database engineering. I am looking forward for this series.
Hey Teiva, would love to join the Coder Cafe Discord to collab or hear some other ideas during this, but the link attached to the emails looks like it’s expired. Any chance of you sending out an updated link?