1. Every community needs a long-term vision if you want to survive. All decisions need to be made in alignment with that vision, and it can change, but it needs to exist. Without a long-term vision, there is no coherent direction on what actions to take, and more importantly, there is no north star according to which the community will align itself.
2. People who truly believe in the vision are the ones who will take the community forward. The community will see many types of people who interact with it, but what differentiates the long-term folks from the passerby is that the long-term folks are believers at heart, and to whatever feasible extent, put the aims of the community before that of their own. The ideal situation is where the aims of the person and the community intersect in harmonious ways.
3. People will remember the fun times much more than they will the work. That's where lasting memories are made. Being serious all the time with no room for goofing off is a recipe for much more of a corporate environment than is suitable for a student community.
4. Build stuff together. It's the single highest ROI activity you can do as a tech community; it's a space for everyone to learn and grow together, and the pride of seeing your own code being used to make things happen is a fantastic thing to experience as a student.
5. Conflict is human and cannot be avoided. It's important to remember that knowing how to manage conflict and work with other people makes you a much better engineer, because at the end of the day, it's other humans that you're working with, no matter how they write their code. Conflict can also teach us a lot about how to build things together, and how to bring together conflicting viewpoints to work towards the same goal.
6. Proof of Work is critical. There is no better testament to how good a community is than the visible work they have, be it blog posts, projects (another reason to build stuff together), or even a well-authored reel.
7. Mutual respect is key. Not having this makes it a toxic environment for everyone.

primary.idx

1. What is an Engine, and DBMS 101
2. Why Clickhouse?
   1. It's just fast
   2. It's fast because -> "to go faster, do less stuff"
   3. Columnarity as an advantage
3. Clickhouse 101
   1. Table Creation and Reads
   2. Parts, Partitions, and Sort Order
   3. No Primary Keys?
   4. Indexing?
      1. Data Skipping Indexes (Bloom Filters)
   5. But What about Concurrency?
      1. Clickhouse uses MVCC -> No Locks, non-blocking inserts
4. Introducing the MergeTree Engine
   1. Why is it called MergeTree?
   2. What's the file structure like?
      1. Sparse Indices
      2. Parts
      3. Marks
   3. The Basic Principle: merging parts in the background
      1. Variants -> CollapsingMergeTree, SummingMergeTree
5. Why are Writes Fast?
   1. Writing as parts
   2. Compression
   3. Wide and Narrow Layouts
   4. Mark Files and Sparse Indexing
   5. Auto-shifting data between HDD and SSD (Multiple Block Devices feature)
6. Why are Reads Fast?
   1. The Primary Key is a superpower
      1. How the Sparse Index helps eliminate granules (to go faster, do less stuff)

Approach - Start off with how data is stored, and how that makes things faster - focus more on organization; data structures exist to make lives easier, choosing the right data structure is essential

What is an Engine?

To understand what an Engine is, we must first understand the components of a database.

Here, we see that a DBMS is nothing but a lot of layers of abstraction that terminate at the storage level. These abstractions inlcude a query parser, a query processor, a compiler, a planner, and a transaction manager. All of these ultimately work towards ensuring that data that has been stored can be queried and manipulated in the most optimal manner possible.

"The storage engine is the part of the DBMS that's responsible for storing, retrieving and managing data in-memory and on-disk" ~ Alex Petrov, Database Internals

So, given the ultimate dependence of the rest of the database on storage and retrieval, the storage engine sometimes supports or implements other features like transactions to give the DBMS developers fine-grained control of what's going on.

The TL;DR is that the storage engine is what supports storing and retrieving data as efficiently as possible, and the rest of the DBMS is built around it, and using it. A DBMS is useless without a good storage engine.

What is Clickhouse?

ClickHouse® is a high-performance, column-oriented SQL database management system (DBMS) for online analytical processing (OLAP).

Clickhouse is what we use here in prod at Bytebeam.

It's a column-oriented DBMS, which means that data is grouped by column, not by row. Through sheer locality and cache-friendliness, this decision to store data by column and not by row makes a massive difference in performance, especially for analytical usecases where column-wise aggregate statistics (mean, median, mode, min, max) and groupings (group by) are extremely common.

OLAP vs OLTP

When we think of a database, we think of postgres or mysql. Clickhouse is similar, but it's an OLAP database, not an OLTP Database. The scope of problems is different here -

1. Dealing with massive datasets -> Billions or Trillions of rows
2. The tables have many, many columns
3. Of these columns, any query needs only a few columns
4. results must be returned in milliseconds or seconds

So, given these constraints, Clickhouse has managed to find a space in which it can make signinficant optimizations with the entire DBMS.

Why is Clickhouse Fast?

To go fast, do less stuff.

Clickhouse makes some pretty impressive claims - "Query a billion rows in milliseconds" is no small matter. There are multiple choices that they took at a design and architecture level to make sure their performance was top-notch, so in reality it is possible for them to query a billion rows in milliseconds.

Let's explore a few of these choices listed here.

What difference does the engine make? Isn't it just a part of the architecture?

So as we've been able to see, the reason clickhouse is able to be so performant is that it has optimizations that reach all the way down to the storage layer and the format. The engine can be considered to be the bottleneck of the entire operation, as it issues the syscalls necessary to persist, fetch, or re-organize data.

You can force Direct I/O (Kernel Page/Buffer Cache Bypass) in Clickhouse with the following

SET min_bytes_to_use_direct_io=1

• clickhouse parallelizes I/O really well. (how?)
• If clickhouse was a car, it would be a drag racer. It doesn't have an optimizer (but somehow has transactional catalogs! Non-ACID Transactions!

Materialized Views can be thought of as synchronous post-insert triggers!! last-point queries in time-series data -> the latest sample of data double-delta encoding for time-series data ~ 99.9% compression ratio

Distributed Joins are not optimized in Clickhouse.

Sharding and replication -> Multi-master, eventually consistent

The MergeTree Engine

• MergeTree Layout consists of indexed chunks of data (every ~8000 or so rows, sparse index) -> Sorted and Compressed. Sprase index (primary.idx) -> multiple granules (distance between rows) .mrk file -> maps granules to a compressed segment inside the .bin file

Insert data into table, atomic merge in the background - you can somehow instantly query them? "Instantly query after insert, but optimize over time"

deletes are expensive! you need to rewrite the entire part.

Why should you know?

The more you understand how the engine works, the faster you can drive it.

The execution Model

Key Components:

• Thread
• Hash Table

Table Creation

CREATE TABLE IF NOT EXISTS sdata (
	DevId Int32,
	Type String,
	MDate Date,
	MDatetime Datetime,
	Value Float64
	-- This is the table engine - there are many variants but can only be one per table
) ENGINE = MergeTree()

-- This is how we'll partition the data (break to pieces in a reasonable way)
PARTITION BY toYYYYMM(MDate)
-- This is how we'll index and sort the data (A Clustered Index)
ORDER BY (DevId, MDatetime)

Insert Processing

INSERT INTO sdata VALUES
(15, 'TEMP', '2018-01-01', '2018-01-01 23:29:55', 18.0),
(15, 'TEMP', '2018-01-01', '2018-01-01 23:30:56', 18.7),

Once this is done, data is assembled in memory (post parsing and planning)

• rows pulled in mem
• "part" of the table created, sorted, and index created, too
• once this is done, we store in the file

Basic parallelization to make inserts run faster. Each thread works on one part.

set max_insert_threads = 4

Q: What is a Part?

Storage Structure

https://altinity.com/wp-content/uploads/2022/05/So-Thats-Why-Its-So-Fast-An-Introduction-to-ClickHouse-Internals-2022-05-16.pdf

A Table consists of multiple Parts A Part consists of a Sparse Index and a set of Columns - Clickhouse uses the primary key as the sort order, same as the clustered index

Section -> 8000~ or so rows in one Granule (Clickhouse is optimized for aggregates, not point lookups)

.mrk -> Mark file, index from primary key to point in compressed .bin file that may or may not contain multiple granules

MergeTree: Because it merges parts in the background!

• Also known as compaction (see tsdbs)
• Also see: Log-Structured Storage (super, super interesting) and LSM Trees

Bigger parts are more efficient!

• PARTITION BY should give you large partitions
• Insert in BULK (~10s of Millions of Rows), keeps merge count low -> Batching is GOOD!

ClickHouse cannot use an index if the values of the primary key in the query parameter range do not represent a monotonic sequence. In this case, ClickHouse uses the full scan method.

For example, the days of the month are partially monotonic sequences.

Sources

• https://www.youtube.com/watch?v=fGG9dApIhDU
• https://www.youtube.com/watch?v=ZOZQCQEtrz8
• https://clickhouse.com/docs/en/intro
• https://clickhouse.com/docs/en/about-us/distinctive-features
• https://clickhouse.com/docs/en/engines/table-engines
• https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree#mergetree
• https://www.youtube.com/watch?v=XpkFEj1rVXg
• https://posthog.com/handbook/engineering/clickhouse/data-storage

This post hopes to walk through some basics of how IO works in Linux. For now, we'll only be considering the interaction of applications with block-based Filesystems, and how they interact with the rest of the subsystems beneath them, more specifically the Block Layer, the Block IO Scheduler, and the request-based device drivers.

The goal here is for both me, and you, dear reader, to understand how the IO subsystem works. To write this post, I will be using two primary sources of information on how this works:

1. Other blog posts and articles about how the IO subsystem works, including, but not limited to, kernel documentation
2. A debugging session that'll trace through what actually goes on when we open, read, and write a file.

Roughly, this is what we're looking at for this blog post.

flowchart TD
	A[Application] -- read(2), write(2), open(2), chmod(2)--> B[VFS Layer]
	B --> C[Block-based FS ext4]
	C --> D[Block Layer with IO Scheduler]
	D --> E[Request-Based Disk Driver]

Design constraints faced by the kernel

Let's consider for a moment that the kernel's job is to make a finite amount of resources (files, memory, CPU) appear as though it's infinite. To this end, the kernel will need to wrap many of the underlying subsystems in abstractions to maintain the "illusion of infinity". For example, the application doesn't know that the memory layout exists a certain way in the actual hardware; it simply assumes that the memory it asks for is given to it, with a far more limited error handling surface than the kernel actually has to deal with.

The point I'm trying to make is that underneath every seemingly simple operation there's a lot of complexity. The application doesn't (and need not) know how all the files on disk are organized relative to each other; it should simply be able to ask to read a file and get it.

System Calls

How does a program make something happen on the host computer? For example, while it is reasonable for us to expect a program to be able to work on its unique functionality, some actions (such as allocating memory, reading/writing from/to files on disk, displaying an image on the monitor) are left in the hands of the Kernel, whose sole function is to perform these duties. This is accomplished by exposing various kernel APIs to the programs, so that when the program wishes to, say, allocate some memory, it can "ask" or "call into" the kernel to do so.

This is known as a system call, or a syscall. These are specialized types of functions that don't exist in the application source code, but are instead run by the kernel. If you've programmed in C before, there's a good chance you've directly used some of these syscalls before - remember malloc?

(more about how system calls work here, if patience permits)

Now, why are system calls important? Doing IO is something we must ask the kernel to do for us, which is why every IO operation happens via a syscall. open, read, write are all syscalls.

Some code

Even in C, we'll usually access these syscalls via a standard library of some sort. Here, it's stdio.h.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main() {

  FILE *myfile = NULL;
  myfile = fopen("hello.txt", "w+"); // <- HERE
  // elided error handling

  char *text = "hello, world!\0";
  size_t stat = 0;

  stat = fwrite(text, sizeof(char), 14, myfile);
  // elided error handling

  // read the same data
  char *newdata = malloc(sizeof(char) * 14);
  // elided error handling

  fseek(myfile, 0, SEEK_SET);

  stat = fread(newdata, sizeof(char), 14, myfile);
  // elided error handling

  size_t cmpres = strncmp(newdata, text, 14);
  // elided error handling

  free(newdata);
  return 0;
}

In the first part that we're concerned about, we'll look at the FILE object, which points to a file descriptor under the hood.

FILE *myfile = NULL;
myfile = fopen("hello.txt", "w+"); // <- HERE

When we call fopen, the IO library stdio does the following:

~~I'm currently on MacOS, so here's the source to apple libc's implementation of fopen: https://github.com/apple-open-source-mirror/Libc/blob/5e566be7a7047360adfb35ffc44c6a019a854bea/stdio/FreeBSD/fopen.c#L60 ~~

I feel it'll be better to trace the code via glibc so we can see what's going on in Linux, not FreeBSD.

https://github.com/torvalds/linux/blob/32a92f8c89326985e05dce8b22d3f0aa07a3e1bd/fs/open.c#L1076 vfs_open

here we first see that we set the path of the new file struct to the path passed in, and then we call into do_dentry_open.

do_dentry_open - this seems like where most of the work is happening: https://github.com/torvalds/linux/blob/32a92f8c89326985e05dce8b22d3f0aa07a3e1bd/fs/open.c#L887

The VFS Layer

A file system is an organization of data and metadata on a storage device. With a vague definition like that, you know that the code required to support this will be interesting.

"Anatomy of the Linux File System" by M. Tim Jones

The bit of software that manages how the data is physically laid out on disk - i.e. which bits in a file map to which physical locations on disk - is called the Filesystem. You may have heard of multiple filesystems like xfs, ext4, btrfs, and so on. All of these filesystems have their own way of representing your files physically. The Linux Kernel provides an abstraction called the VFS to ensure that applications need only one codepath for the Virtual File System (VFS) layer, while the VFS calls into the underlying filesystems. the VFS layer implements multiple syscalls such as open, stat, read, write, and chmod ¹.

The VFS layer also contains two caches, one for dentrys and one for inodes. What are dentry and inode? Both are filesystem objects used to represent items present in the filesystem.

inode

An Inode represents a file on the disk (We need to keep in mind that in linux, directories are files too, just special kinds of files). They store metadata about a file, as well as the locations of the disk blocks that make up the file. The inodes are stored on the disk. They notably do not contain the filename. Each inode has an inode number, which is unique within a filesystem.

The inode contains (by a number of means) a list of disk block locations that comprise the file. An important point to consider is that the same way the inode for a file points to other (data) blocks on disk, the inode structure can be reused to be a directory that points to multiple other inode blocks on disk.

https://www.linfo.org/inode.html

dentry

so, given that we don't have filenames defined yet, and directories are just files, how do we figure out what our filesystem looks like? The answer to this is the dentry. These are in-memory objects that are created on boot by the kernel walking the filesystem, which map directories to paths, such that if you would like to look at all the folders on your disk, you simply need to look at all the dentries in memory.

So, the VFS layer caches both inodes and dentries. This allows us to speed up a number of operations on disk.

is there a buffer cache as well?

The Filesystem

The filesystem determines how data is ultimately organized on disk and how it's updated. It follows that the type of filesystem you use (and its many features) will end up determining application performance and other run-time characteristics. Some common filesystems that are used today are ext3, ext4, xfs and btrfs.

Each filesystem has a superblock which has a list of all the inodes present on the filesystem. This superblock is usually located at exactly the same location in every partition/disk.

https://blogs.oracle.com/linux/understanding-ext4-disk-layout-part-1 GDT? BDT?

The Block Layer

The linux kernel must manage many block devices. It is possible for a physical disk to have multiple partitions (each can be considered a disk in its own right), each with its own partition on it - this means that though there's a very real hardware bottleneck that exists in the bandwidth of the disk, the kernel must schedule block IO in such a way that the bandwidth of the disk is distributed "Fairly". To that end, there are also multiple IO Schedulers that exist to manage all the inflow of IO onto the disk. Some schedulers are the kyber scheduler, mq scheduler, and deadline scheduler. https://www.cs.cornell.edu/courses/cs4410/2021fa/assets/material/lecture24_blk_layer.pdf

BIOs -> Block IOs

what is the device mapper?

[superblock ][inode section][data section]

What's Meilisearch?

Meilisearch is a "powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box".

To put it simply, Meilisearch allows you to implement fast search at scale, and does it in Rust. Take a look at this GIF and see it in action. It's safe to say it blew my mind the first time around.

What was the Issue?

• Find the issue itself here.
• Find the Pull Request to close this issue here.

Meilisearch has a browser-accessible dashboard that helps you test the implementation as and when you install it. This dashboard is usually how most people first interact with Meilisearch, and it's where the demo GIF comes from. Given the above, it's easy to understand why it's a critical user-experience component of the application.

I, on the other hand, was (and still am) a curious rustacean looking to better my knowledge, so I was naturally interested when I saw this issue open with the good first issue label. I jumped at the chance to figure this out, and then I cloned the repository. It's the largest codebase I've interacted with, and I was naturally somewhat intimidated by what was going on.

This repository didn't deal with the core implementation of search - that's taken care of by milli - to quote, "The code in this repository is only concerned with exposing the HTTP API, managing multiple indexes, and handling the update store".

After I cloned it, I got started with trying to fix this error. There were no error messages while this project ran, which meant that this bug would be harder to sniff out.

The Debugging Process

All I had to start with were a bunch of errors in the browser console.

From my experience with frameworks such as Django and Flask, This usually meant some file wasn't being served properly. Here's what I did to dig a little deeper -

1. I Ran the test suite, and All the tests passed

   This was simple enough - all I had to do was run

   $ cargo test
   

   Rust has some pretty neat features baked directly into the toolchain, which helps you write tests pretty easily. As you see above, it's also super easy to run a test suite.

2. Investigated the built output

   I then proceeded to build the project, for which all I had to do was run

   $ cargo build
   

   I found the build output at /meilisearch-http/target/rls/debug/build/meilisearch-http-a6e2793cb../out/mini-dashboard/. This consisted of a bunch of static files pulled from an external source, which were then served using the Actix Web Framework.

   Since there were a bunch of static files wired up, I used the npm package serve to run a temporary web server here, which served all files correctly.

   # if you haven't got it installed already
   $ npm install -g serve # installs this globally
   $ serve
   

   This confirmed my initial suspicion that it wasn't a problem with the dashboard itself, but rather with how it was being served.

3. I then Looked into how the server was setup.

   At this point I used simple print debugging to ensure that the paths were generated correctly, along with their MIME types. This, too, is relatively simple - you can use the println!() macro to easily print the resource in question.

   All seemed to be in order. I also tried using the format! macro to prepend a slash onto the file path for each file, but this didn't make a difference at all.

4. At this point I noticed that the index.html file was the only file being served correctly.

   Taking a look into the route configuration, I noticed that the route for the index file was configured like this -

   config.service(web::resource("/").route(
       web::get().to(move || HttpResponse::Ok().content_type(mime_type).body(data)),
   ));
   

   as opposed to

   scope = scope.service(web::resource(path).route(
       web::get().to(move || HttpResponse::Ok().content_type(mime_type).body(data)),
   ));
   

   Given I'm not too familiar with Actix, I took the approach of registering the routes for the static files the same way as the index.html file, replacing the above block with the following to serve static files -

   config.service(web::resource(path).route(
       web::get().to(move || HttpResponse::Ok().content_type(mime_type).body(data)),
   ));
   

This worked! localhost:7700 now produced the dashboard properly.

Why did this change work?

Given I haven't explored Actix in depth, I'm probably not the best person to answer this question, but if I had to take a guess, I'd say it has to do with how Actix internals decided route registration for static assets should take place. It also probably calls for special handling of the index page at /, and this caused the issue.

I'll be back with more information on this when I've learnt more!

Wrapping Up

This was my first contribution of any sort to a major open-source project, and I'm glad it was a Rust project! It's a really cool language, and I'm glad I got to learn more about it. The maintainers were super nice, and this made me feel welcome! I was happy when this PR got merged, and have, since then, felt a little more confident about my ability to navigate large codebases which I have no familiarity with. This was fun!

I look forward to contributing to more, larger open-source projects!

I'm very passionate and curious about the data systems space as a whole, with specific interest in distributed databases, streaming computation, dataflow systems, and storage engines, with the occasional detour into things like formal verification, and then some. I'm also interested in keeping up with the research happening in said spaces - that should give you a good enough idea of what's to come as a part of this newsletter.

I'm writing this newsletter to share the things I know (admittedly not much) and the things I come across (much more than the latter), along with possibly a deep dive or some small explanation of a new thing I've learnt, every two weeks or so. I'm doing this in the hope that there's at least one person who learns something new every two weeks; I find that to be a non-negotiable given that my curiosity has largely been fuelled by the work of folks who chose to share what they knew and learnt, and this is my way of giving back.

That being said, I'm still largely figuring out what this newsletter is going to look like, the cadence, the content, so much of it is undetermined - you're seeing the transaction before it commits here xD I'm sure I'll end up changing a bunch of stuff, but what isn't going to change is that I'm going to be writing about the things I've been interested in.

I have no deep dive for you folks this month, as I keep getting nerd-sniped by nice questions that I come across whilst writing it; for example, why do LSM Trees have levels? More on that in the next edition.

A Tribute to some of my inspirations

It seems fair that I start this off by talking about some of the talks and lectures that inspired me the most.

CockroachDB Internals

This was the first distributed database I learnt about. I was so fascinated by everything in this video - distributed transactions, consensus, a distributed KV layer, consistency semantics, consensus groups, sharding... this melted 19-year-old me's brain, and gave me a burning curiosity that brought me to where I am today. No further comments from me, this is a fantastic watch!

Video Link

Deterministic Simulation and Testing in FoundationDB

I came across this talk before Deterministic Testing/Simulation was made a popular idea by Antithesis (founded by the speaker!), Turso, and friends; I find FoundationDB to be one of the most interesting databases out there for a number of reasons (it has a mini transaction engine in every client, among other things), and this talk dived really deep into how the team worked really hard to find and eliminate what would have otherwise been nightmarishly hard bugs to catch. I found it really cool that they had an array of nodes that they just left running to catch some of those particularly crazy bugs.

Video Link

RocksDB Internals, explained by Dhruba Borthakur

Having worked on LSM Trees at work, I have a fraction of an insight into how hard it is to build performant and correct storage systems. This talk by Dhruba Borthakur, one of the earliest folks at RocksDB, now relatively ubiquitous as an embeddable and performant Key-Value store, talks about embedded KV Databases are necessary, explains the LSM Tree architecture, talks about some of the problems with LevelDB and then introduces RocksDB. Truly a fantastic tour of the space not just from the perspective of a systems engineer but also from that of someone who would consume the store in their product.

Video Link

SkyPlane (NSDI 2023)

This project aims to solve the problem of slow and expensive data egress (for many reasons) from object storage offerings that major cloud providers offer by rather craftily building an ad-hoc overlay routing network on top of compute nodes on said providers, exploiting parallel VMs and parallel connections to make the data transfer much faster. Admittedly, I don't understand this very well, but it's very cool.

Video Link

Time, Clocks, and the Ordering of Events in a Distributed System - Leslie Lamport

Paper: https://lamport.azurewebsites.net/pubs/time-clocks.pdf

This is one of my favourite research papers of all time, right alongside the Mapreduce paper.

Lamport starts by considering the problem of ordering events in a distributed system, defined more succinctly as a collection of processes that are spatially separated, that communicate with each other by sending messages. Ordering is important, and is often the source of correctness in many systems. This paper walks through the notion of "Logical Time" - time that doesn't rely on (at the time) inaccurate physical clocks - and also provides an algorithm to convert a partial ordering of events to a total ordering, deriving bounds on clock drift.

It's a dense paper, with a lot of invariant-based reasoning, but it's so worth it - especially the part where they explain the design of a distributed, leaderless mutex lock.

And that's a wrap

Thank you for reading! I hope at least one of these videos has brought you some joy, and most importantly, curiousity!

As always, I'm open to a conversation or feedback about any of this at ani dot rowjee at gmail dot com. Stay tuned for the next edition, where I'll be talking about LSM Trees in more detail.

Signing off,
Anirudh

I was reading my friend, Anant Thazhemadam's blog, and happened to come across this line.

"This blog is for me. By that, I mean it is for me, and me only."

I realized that i've spent most of my time with this blog as a platform to showcase who I am. While that's definitely the aim I've got in mind when I work on it, I'm realising it quickly became a place for me to put content out for someone else to see. To that end, I started considering the blog to be a place to show what I wanted to be, not what I am.

Till date, every single thing I wrote on this blog was written keeping in mind the perspective of someone who's looking to hire me, or work with me. While this is important in terms of building a brand, it's absolutely terrible for keeping this home on the internet in sync with who I am and what I do.

No more!

This blog is going to

• firstly, not be called a blog anymore, because that isn't what I want to limit it to - It's going to be my home on the internet. I'm going to post a lot more about my life - pictures I like, for instance, or poems that I wrote and like.

• secondly, not limit itself to only CS Stuff. I'm passionate about Computer Science, Software Development and Technology, but that isn't all I'm passionate about. I've got other interests, and write about things that aren't necessarily this. It follows that my space on the internet reflects this, and puts these other things out there for the rest of the internet to see, too.

So here's what's changing, apart from the interface of some components of this blog -

I'm going to stop writing posts specifically for the blog.

Anything that goes up on the blog will be a side effect of what i'm currently doing. Sure, this may make the space a little less technical at times, but that's the entire point - I'm not (and should not be) writing it to meet some criteria of what a "good" blog looks like. It might also change the frequency with which I'm trying to post.

This blog is my home on the internet. It should reflect what I'm up to, and what I'm working on, not necessarily only what other people are looking to understand out of the blog. It's going to be a lead indicator of what I'm working on, and a lag indicator of who I am.

In the old days, back when every single bit of information on the internet wasn't commoditized, people's homepages were a true window into who they were. It represented their thoughts, their ideas and their opinions (yes, opinions!).

This post is called reclaim, because I'm reclaiming this space to be my own.

Skiplists it is. public declaration: I'm going to write a blog post titled "understanding and implementing skiplists" and publish it on the 1st of October, and It's going to be good! https://t.co/LQ5VhKqsx8

— Anirudh Rowjee @ override.bsky.social (@AnirudhRowjee) September 7, 2024

I gave myself 22 days to build a skiplist and write about it, so here we are. This is in no ways a perfect or even a good implementation; I've tried to be as hacky as possible, getting the bare minimum core functionality working while ignorning just about everything else.

You also have access to my C++ code¹ and the original paper², which contains most of the diagrams you'll see on this post. I've tried my level best to not look at other implementations before I implemented this, so the spirit of "implemented straight from the paper" is retained to a fair extent.

What are Skiplists?

A skiplist is a probabilistic, associative data structure based on linked lists. From a functionality point of view, it resembles a hashmap, or a dictionary. The best way to think about it is that it's an in-memory key-value store.

Formally, It allows the following operations (and you can swap out the strings for bytes, etc, any things that you can compare)

1. upsert(string key, string value)
2. search(string key) -> optional<string value>
3. scan(string start, string end) -> Vector<pair<string, string>>
4. delete(string key)

How did we get here?

I have a bad problem with explaining everything - and though it makes for a good read, I'm trying to be more succinct so we can cover more stuff!

Consider the problem of searching over a set of keys - a collection, if you will. Attaching a value to the key is trivial.

The primitive linear search algorithm for almost any data structure (linked list, array, etc) is almost always O(n), which means it grows asymptotically at the same rate as the input. If we know all data beforehand, we can sort it, and then we can leverage algorithms such as binary search to give us O(log_2(n)) complexity, which work by successively reducing the search space by a factor of two (often at an additional O(n * log_2(n)) cost to sort the input).

Binary Search, Wikimedia Commons

However, we rarely know all inputs beforehand - we can't just dump the inputs in the order they came in, because that would break the sortedness invariant we have in order to achieve O(log_2(n)) complexity on search.

The ideal data structure then is something that supports maintaining the "sorted-ness" invariant with little interference from the user, and that also makes it easy to handle the following operations - insert, update, delete, scan - with relatively low asymptotic complexity.

The most ideal data structure for this task, and indeed, the choice of implementation for most languages and their standard implementations, is the Red-Black Tree, or some other form of self-balancing binary search trees. These data structures usually use a heuristic to approximate the "balanced-ness" of the tree - usually called balance factor - and often rely on complex rebalancing algorithms that must shuffle multiple pointers in place to accommodate for this.

Red-Black Tree, Wikimedia Commons

Balanced Binary Search Tree Rotation, Wikimedia Commons

What if we could trade complexity for correctness? What if we could settle for an average case O(log(n)) with a worst case O(n), as opposed to the guaranteed average and worst case O(log(n)) with Binary Search? This is exactly the guarantee that skiplists make.

What does a skiplist look like?

Think of a singly linked list where each node has more than one pointers to other nodes organized in levels, where the highest level (say, 5) represents the largest number of nodes skipped, and the lowest level represents the lowest number (0) of nodes skipped (i.e. points to its immediate neighbour).

In this case, the 4th level connects every 8th node, the 3rd level connects every 4 nodes, the 2nd level connects every 2 nodes, and the 1st level connects every successive node.

Primitive example of a singly linked list with multiple forward pointers

A Skiplist can be thought of as a modification of a singly linked list that is maintained in sorted order, with the change that each node holds a fixed, randomly determined number of pointers to successive nodes, organized in the same paradigm of levels.

A Skiplist, from the paper

When we combine the sorted nature of the nodes in the a singly linked list (sorted by key, for example), and the idea of levels that allow you to skip some series of nodes, you start to see how it all falls into place; You can start at the highest level and make the largest jumps, successively reducing the search space as we move down the levels, and finally, arriving at the target node. This is a skiplist.

search(key k):
	Start at the highest level
	repeat until we are at the lowest level:
		repeat until the next element key is greater than the search key
			go to the next element
		move down one level

An Intuition for Search

The best intuition I've heard for this is from Prof Srini Devas from MIT 6.046J³. In this case, we consider each node to be a subway/train station, and we consider pointers to be the train routes between them.

The Purple Line of the Bengaluru Metro, made into a skiplist

In this scenario, we consider an arrangement of multiple rail lines (and thus, trains) plying between the same stations. Each station can have multiple trains!

Let us consider that the time taken to travel between any two stops is constant, regardless of how far they are. If I need to get from KSR Station to Central College Station, I am spoilt for choice; I have three train lines to choose from. If I want to get from KSR Station to Cubbon Park Station, I have two choices - I can either use the red line (no stops) or the orange, green, and purple lines (one stop). Clearly, the approach for the red line wins!

Lastly, consider a scenario where I have to go from KSR Station to Trinity Station. It might appear that the default path is 2 stops (KSR -> Central College -> Cubbon Park -> Trinity) - but in reality, I can do this in 1 stop by going from KSR -> Cubbon Park (via the red line), and then going from Cubbon Park -> Trinity (via the purple line). The key learning is that we can greedily adjust our strategy to have the least number of stops, provided that some sort of balance is maintained that doesn't negatively bias this whole process.

Who decides which lines connect which stations?

That's an extremely important question, and it's a critical part of what makes skiplists so successful. Keep that question in mind - this is also where probability comes into the picture.

The Math

As we discussed before, we rarely have all the data in our dataset present beforehand, which is why our skiplist must tolerate insertions, updates, and deletions, all while maintaining a reasonably low search complexity. In this reality, we can't assume that we know how many nodes there will be or where they'll be positioned, which is why deciding the level of a skiplist node is not trivial.

To ensure that, probabilistically, we have the highest change of skipping the most nodes in our search process, we must reduce the number of nodes in each level as we move from the lowest to the highest level. We can do this by

1. Randomly setting the level of a node at insert time and
2. controlling the rate of increase of the level of a node

To control the rate of increase, we use a fraction p - As Wikipedia puts it,

"... where an element in layer i appears in layer i + 1 with some fixed probability p"

And as the paper puts it,

"To get away from magic constants, we say that a fraction p of the nodes with level i pointers also have level i+1 pointers."

Choosing p well is a matter of tuning, and the paper has some recommendations to make, given that it basically trades off space for accuracy.

The paper has a nice proof that fashions search instead as "climbing out of a list", and proves that search complexity is equal to O(log(n)). Wikipedia has a nice proof⁴ of the search cost being O(log(n)) as well.

Maintenance of a skiplist

Given that we understand that the skiplist is just an enhancement of a sorted linked list that has been augmented with more pointers, we see that every algorithm that modifies the skiplist (insertion, deletion, update) has the same rough structure:

mutate(skiplist):
	find element to modify / insert after
	update existing element OR create/delete the element
	maintain the chain of pointers

The first two steps are rather straightforward: for 1, we use the search algorithm we just learnt about, and 2 is just memory allocation/deallocation.

When it comes to pointer maintenance, we use a vector of pointers to the nodes right before the current node, called the update vector, to make our lives easier. The Update Vector holds all the predecessor nodes that will need to be updated if a new node is added, and allows us to maintain "pointer continuity" in that sense.

Advantages over other data structures

• Skiplists don't require expensive rebalance operations (see red-black trees)
• Skiplists make it very easy to do sequential scans on data (see red-black trees)
• Skiplists can be more easily modeled as concurrent data structures using atomic pointer operations ⁵

Implementing a skiplist

Fair warnings:

1. Using clang is frustrating sometimes. And Don't get me started on CMake.
2. The paper itself is 1-indexed, and I hope all your code is 0-indexed - this tripped me up for a while and caused some really funny segfaults, so be careful lol
3. Everything in every class is public - I know, this is very C-style programming, but I couldn't be bothered to write the appropriate accessors.

Defining the skiplist node

class SkiplistNode {

public:
  std::vector<SkiplistNode *> links;

  // Constructor
  SkiplistNode(int current_level, std::string key, std::string value)
      : current_level(current_level), key(std::move(key)),
        value(std::move(value)) {

    links = std::vector<SkiplistNode *>(current_level);
    for (int i = current_level - 1; i >= 0; i--) {
      links[i] = nullptr;
    }
  }

  void DUMP() {
    std::cout << fmt::format("SkiplistNode[{}] [{}:{}] AT {} ", current_level,
                             key, value, fmt::ptr(this))
              << std::endl;
    for (int i = current_level - 1; i >= 0; i--) {
      std::cout << fmt::format("\tNode at level {} pointing to {}", i,
                               fmt::ptr(links[i]))
                << std::endl;
    }
  }

  int current_level;
  std::string key;
  std::string value;
};

Defining the Skiplist Class

I decided to explore std::optional<T> this time as a cure for my rust hangover; It works very nicely! the lack of a Result<T> equivalent made me sad but I hear that that's in the works too for C++23 with std::expected<T, E>. I've made up for this with std::pair<T, SkiplistError>, leading to some fairly go-like C++ code.

class SkiplistError {

public:
  enum ErrorVariant { BAD_ACCESS, ALLOC_FAIL, KEY_NOT_FOUND, NOERR };
  ErrorVariant e = ErrorVariant::NOERR;
  std::string message;

  SkiplistError(ErrorVariant e, std::string message = "")
      : e(e), message(message){};

  operator bool() { return e != NOERR; }
};


class Skiplist {

public:
  // constructor
  Skiplist(int max_level);

  // Get the value associated with a particular key
  std::optional<std::string> Search(std::string Key);

  // Insert a key-value pair
  std::pair<std::string, SkiplistError> Insert(const std::string &Key,
                                               const std::string &Value);

  // Delete a key and get the value associated with it on successful delete
  std::pair<std::string, SkiplistError> Delete(std::string Key);

  // Do a full scan of the skiplist
  // TODO see if we can replace this with an iterator
  std::pair<std::vector<std::pair<std::string, std::string>>, SkiplistError>
  Scan();

  // Find the point to insert a new skiplist node
  std::pair<std::pair<SkiplistNode *, std::vector<SkiplistNode *>>,
            SkiplistError>
  identifyPredecessorNode(std::string key);

  void DUMP();

  ~Skiplist();

  SkiplistNode *START;
  SkiplistNode *END;
  int max_level;
  int current_max_level;
  // p: a tunable factor for the number of elements you want in the skiplist
  float p;
  std::mt19937 rng;
  std::uniform_real_distribution<double> distribution;

  int getRandomLevel();
};

Constructing a skiplist

This is the test that we wanted to pass:

TEST(SkiplistTest, test_init) {
  // Initialize a Skiplist
  const int max_level = 5;
  auto sl = Skiplist(max_level);
  for (int i = 0; i < max_level; i++) {
    ASSERT_EQ(sl.START->links[i], sl.END);
  }
}

and here's the constructor:

Skiplist::Skiplist(int max_level) : max_level(max_level) {
  p = 0.5;
  rng = std::mt19937(std::time(nullptr)); // fixed seed as of now
  distribution = std::uniform_real_distribution<double>(0.0, 1.0);
  // Initialize the sentinel nodes
  START = new SkiplistNode(max_level, "START_KEY", "START_VALUE");
  END = new SkiplistNode(max_level, "END_KEY", "END_VALUE");
  // Connect the start and end nodes
  for (int i = 0; i < max_level; i++) {
    START->links[i] = END;
  }
}

Here, START and END are special nodes, which are the sentinel nodes of the skiplist. In all honesty, it would probably be "cleaner" to use a nullptr to terminate the node pointers, but I felt like having a dedicated END node would be a better option. This would also help maintain the invariant that there will never be a nullptr in the search path, as we will see later.

Implementing Insertion

Given that insertion and search are two really tightly coupled algorithms (you need search to implement insertion, but you also need insertion to be able to test search?), I was forced to implement both at once.

Here's the test we wanted to pass (and there's another one that tests search later as well).

TEST(SkiplistTest, test_insert) {

  auto linear_search = [](std::string key,
                          Skiplist *sl) -> SkiplistError::ErrorVariant {
    auto curr_node = sl->START;
    while (curr_node != sl->END) {
      if (curr_node->key == key) {
        return SkiplistError::ErrorVariant::NOERR;
      } else {
        curr_node = curr_node->links[0];
      }
    }
    return SkiplistError::ErrorVariant::KEY_NOT_FOUND;
  };

  // Check the monotonicity of the keys
  auto monotonicity_check = [](Skiplist *sl) -> SkiplistError::ErrorVariant {
    auto currentNode = sl->START->links[0];
    std::string prevText = "";
    std::string currentText = "";
    while (currentNode != sl->END) {
      prevText = currentText;
      currentText = currentNode->key;
      if (!(currentText > prevText)) {
        return SkiplistError::BAD_ACCESS;
      }
      currentNode = currentNode->links[0];
    }
    return SkiplistError::ErrorVariant::NOERR;
  };

  const int max_level = 5;
  std::cout << fmt::format("initializing skiplist with level {}", max_level)
            << std::endl;
  auto sl = new Skiplist(max_level);

  std::vector<std::tuple<std::string, std::string>> kvPairs = {
      {"hello", "world"},   {"something", "else"},
      {"enter", "sandman"}, {"the struts", "could have been me"},
      {"hello", "world2"},  {"END_KEY", "SYSTEM BROKEN!!!"}};

  for (auto [key, value] : kvPairs) {
    // Insert an element and search for it
    auto [retval, err] = sl->Insert(key, value);
    ASSERT_EQ(err.e, SkiplistError::NOERR);
    sl->DUMP();
    // Do a simple linear search
    ASSERT_EQ(linear_search(key, sl), SkiplistError::NOERR);
    // Ensure that monotonicity is maintained
    ASSERT_EQ(monotonicity_check(sl), SkiplistError::NOERR);
  }
}

Here's the insertion code, beginning with identifying the predecessor node. This code is common across all mutating functions, and will be reused in other places, so I've made it a function of its own; it returns the pointer to the predecessor node or the node itself, and the update vector of nodes that we used to reach the predecessor node.

std::pair<std::pair<SkiplistNode *, std::vector<SkiplistNode *>>, SkiplistError>
Skiplist::identifyPredecessorNode(std::string key) {

  // Initialize the update vector
  auto update = std::vector<SkiplistNode *>(max_level, nullptr);

  // See if the node already exists
  auto current_node = START;
  auto next_node = START;

  auto max_search_level = max_level - 1;
  for (int i = max_search_level; i >= 0; i--) {
    // Check if the next node in the level has a key that comes before our
    // search key

    // See if the next node is a sentinel element
    next_node = current_node->links[i];
    while (next_node->key < key && next_node != END) {
      current_node = next_node;
      next_node = current_node->links[i];
    }
    update[i] = current_node;
  }
  current_node = current_node->links[0];
  return std::make_pair(std::make_pair(current_node, update),
                        SkiplistError(SkiplistError::ErrorVariant::NOERR));
}

And finally, the insertion code itself:

// This is called insert but it has upsert semantics
std::pair<std::string, SkiplistError>
Skiplist::Insert(const std::string &key, const std::string &value) {

  // Figure out where to insert the node: this is either the node with the same
  // key, so we can update the value, or we found the node right before the
  // insertion point so that we can insert after it
  auto [meta, error] = identifyPredecessorNode(key);
  if (error.e != SkiplistError::NOERR) {
    return std::make_pair("", error);
  }
  std::cout << "Predecessor Found!" << std::endl;

  auto [current_node, update] = meta;
  if (current_node->key == key) {

    // If the key exists at the current node, update its value
    std::cout << fmt::format("BAZINGA! Key {} found with value {}",
                             current_node->key, current_node->value)
              << std::endl;
    auto oldVal = current_node->value;
    current_node->value = value;
    return std::make_pair(value, SkiplistError::NOERR);

  } else {

    // If they key doesn't exist at the current node, insert it
    std::cout << "Node not found" << std::endl;
    // The node doesn't exist;
    // Construct a new node
    int level = getRandomLevel();

    // WARN
    // Not doing this rn: need to update the max level of the list if this
    // happens I doubt it will but the paper says we need to do it so ¯\_(ツ)_/¯
    if (level > max_level) {
    }

    auto new_node = new SkiplistNode(level, key, value);
    for (int i = 0; i < level; i++) {
      new_node->links[i] = update[i]->links[i];
      update[i]->links[i] = new_node;
    }

    // Update all the pointers in the reachability chain to reach this node
    return std::make_pair(value, SkiplistError::NOERR);
  }
}

As we discussed before, the level of a new skiplist node is governed by a random number generator. The p factor here is extremely crucial to the "balancing" of your skiplist;

int Skiplist::getRandomLevel() {
  int level = 1;
  while ((distribution(rng) < p) && (level < max_level)) {
    level += 1;
  }
  return level;
}

Implementing Search

Test to pass:

TEST(SkiplistTest, test_insert_and_search) {
  const int max_level = 5;
  std::cout << fmt::format("initializing skiplist with level {}", max_level)
            << std::endl;
  auto sl = Skiplist(max_level);
  sl.DUMP();

  // Load some data into the skiplist
  std::vector<std::tuple<std::string, std::string>> kvPairs = {
      {"hello", "world"},
      {"something", "else"},
      {"enter", "sandman"},
      {"martin garrix", "under pressure"},
      {"the smiths", "please please please let me get what I want"},
      {"the struts", "could have been me"},
      {"hello", "world2"}};
  for (auto [key, value] : kvPairs) {
    auto [retval, err] = sl.Insert(key, value);
    ASSERT_EQ(err.e, SkiplistError::NOERR);
  }

  // Search for elements in the skiplist
  std::vector<std::tuple<std::string, std::optional<std::string>>> testcases = {
      {"something", "else"},
      {"enter", "sandman"},
      {"the struts", "could have been me"},
      {"martin garrix", "under pressure"},
      {"this isn't there", std::nullopt},
      {"the smiths", "please please please let me get what I want"},
      {"hello", "world2"},
      {"nonexistent", std::nullopt}};
  for (auto [key, searchResult] : testcases) {
    ASSERT_EQ(sl.Search(key), searchResult);
  }
}

And the algorithm, most of which has been taken from the identifyPredecessorNode function, both of which are based on the paper:

std::optional<std::string> Skiplist::Search(std::string Key) {
  std::cout << "Searching for -> " << Key << std::endl;
  auto current = START;
  auto max_search_level = max_level - 1;
  for (int i = max_search_level; i >= 0; i--) {
    while (current->links[i]->key < Key && current->links[i] != END) {
      std::cout << fmt::format("Exploring key -> {}", current->links[i]->key)
                << std::endl;
      current = current->links[i];
    }
  }
  current = current->links[0];
  if (current->key == Key) {
    return std::optional<std::string>{current->value};
  } else {
    return std::nullopt;
  }
}

Implementing Delete

This also followed the same pattern as the insert operation as before, just that it now frees memory as well as deleting everything else.

Here is the test we need to pass:

TEST(SkiplistTest, test_delete) {
  const int max_level = 5;
  std::cout << fmt::format("initializing skiplist with level {}", max_level)
            << std::endl;
  auto sl = Skiplist(max_level);

  // Load some data into the skiplist
  std::vector<std::tuple<std::string, std::string>> kvPairs = {
      {"hello", "world"},
      {"something", "else"},
      {"enter", "sandman"},
      {"martin garrix", "under pressure"},
      {"the smiths", "please please please let me get what I want"},
      {"the struts", "could have been me"},
      {"hello", "world2"}};
  for (auto [key, value] : kvPairs) {
    auto [retval, err] = sl.Insert(key, value);
    ASSERT_EQ(err.e, SkiplistError::NOERR);
  }
  // Delete some elements
  std::vector<std::tuple<std::string, SkiplistError::ErrorVariant>> testcases =
      {{"hello", SkiplistError::NOERR},
       {"enter", SkiplistError::NOERR},
       {"this will break", SkiplistError::KEY_NOT_FOUND}};

  for (auto [key, err] : testcases) {
    std::cout << fmt::format("Deleting {}", key) << std::endl;
    auto [retval, reterr] = sl.Delete(key);
    sl.DUMP();
    ASSERT_EQ(reterr.e, err);
  }
}

And here's the implementation:

std::pair<std::string, SkiplistError> Skiplist::Delete(std::string Key) {
  std::string oldVal;
  // Figure out where to delete the node: this is either the node with the same
  // key, so we can update the value, or we found the node right before the
  // insertion point so that we can insert after it
  auto [meta, error] = identifyPredecessorNode(Key);
  if (error.e != SkiplistError::NOERR) {
    return std::make_pair("", error);
  }
  auto [nodePtr, update] = meta;
  if (nodePtr->key == Key) {
    // We have found the right node to delete
    // update all the necessary pointers
    for (int i = 0; i < max_level; i++) {
      if (update[i]->links[i] != nodePtr) {
        break;
      }
      update[i]->links[i] = nodePtr->links[i];
    }
    oldVal = nodePtr->value;
    delete nodePtr;
    // TODO update overall list level
    return std::make_pair(oldVal,
                          SkiplistError(SkiplistError::ErrorVariant::NOERR));
  } else {
    return std::make_pair(
        "", SkiplistError(SkiplistError::ErrorVariant::KEY_NOT_FOUND));
  }

  return std::make_pair("",
                        SkiplistError(SkiplistError::ErrorVariant::BAD_ACCESS));
}
}

Implementing Scan

This one is pretty simple. We only need to read all key-value pairs into an array. There's probably a better way of doing this, but this will suffice for now.

The test we want to pass:

TEST(SkiplistTest, test_fullscan) {
  const int max_level = 5;
  std::cout << fmt::format("initializing skiplist with level {}", max_level)
            << std::endl;
  auto sl = Skiplist(max_level);

  // Load some data into the skiplist
  std::vector<std::pair<std::string, std::string>> kvPairs = {
      {"something", "else"},
      {"enter", "sandman"},
      {"martin garrix", "under pressure"},
      {"the smiths", "please please please let me get what I want"},
      {"the struts", "could have been me"},
      {"hello", "world2"}};
  for (auto [key, value] : kvPairs) {
    auto [retval, err] = sl.Insert(key, value);
    ASSERT_EQ(err.e, SkiplistError::NOERR);
  }

  // Copy over the kv pairs
  std::vector<std::pair<std::string, std::string>> kvPairsSorted = kvPairs;

  std::sort(kvPairsSorted.begin(), kvPairsSorted.end(),
            [](std::pair<std::string, std::string> &a,
               std::pair<std::string, std::string> &b) -> bool {
              return std::get<0>(a) < std::get<0>(b);
            });
  std::cout << "Printing out sorted list: " << std::endl;
  for (auto k : kvPairsSorted) {
    std::cout << fmt::format("Key {} Value {}", std::get<0>(k), std::get<1>(k))
              << std::endl;
  }

  auto [res, err] = sl.Scan();
  ASSERT_EQ(err.e, SkiplistError::NOERR);
  ASSERT_EQ(res, kvPairsSorted);
}

and here's the implementation:

std::pair<std::vector<std::pair<std::string, std::string>>, SkiplistError>
Skiplist::Scan() {
  std::vector<std::pair<std::string, std::string>> answer = {};
  auto current_node = START->links[0];
  while (current_node != END) {
    answer.push_back(std::make_pair(current_node->key, current_node->value));
    current_node = current_node->links[0];
  }
  return std::make_pair(answer, SkiplistError(SkiplistError::NOERR));
}

Destructor

There has to be a better way to do this, but I wanted to be safe.

Skiplist::~Skiplist() {
  std::cout << "Destructing Skiplist" << std::endl;
  auto [res, err] = Scan();
  if (err.e != SkiplistError::NOERR) {
  } else {
    for (auto [k, _] : res) {
      auto [res1, err1] = Delete(k);
      if (err1.e != SkiplistError::NOERR) {
        std::cout << fmt::format("Failed to delete key {}", res1) << std::endl;
      }
    }
  }
}

Notes

• you can find the code for DUMP, as well as the CMakeLists.txt file on Github.
• My rule of not looking at another implementation might have harmed me more than it helped me!
• only a few null pointers were harmed in the making of this post.

Acknowledgements

I'd really like to thank Phil Eaton, whose practice of writing monthly technical blog posts has been a long-standing inspiration of mine, and thanks to whose encouragement this blog post exists.

I would also like to thank Navin Shrinivas for being my accountability buddy throughout this process, and Achyut Yogesh Sosale, Kiran Rajpurohit, Siddarth Tewari, and Anirudh Sudhir for their suggestions on how I could make this post better. Thank you folks so, so much!

And lastly, to everyone on twitter who has been listening along on my journey and supporting me with encouraging comments and feedback - thank you!

And thank you, dear reader, for reading :D if you have any feedback (suggestions, comments, corrections), you can find me on twitter or email.

References

levelDB doesn't have any external synchronization primitives - it's interesting to note this as a design primitive, where the write path handles queueing/waiting for access to the internal store. I found it fascinating that your current write is just in queue waiting to get access, and this happens without the user being aware of any queueing.

fsync is optional - the write options have a way to let you skip fsyncing the write, leaving it potentially stuck in the pagecache. I don't think this is a good thing, and I'm struggling to see a usecase for when I might want to skip fsync.

there are two memtables - this is common and back when it was designed i'm sure IO slowness would have warranted it, because you usually have two memtables when flushing a memtable to disk takes a lot of time. In the grand scheme of things it still does - one in-memory IO vs one disk IO.

the WAL is represented as a log, which has its own internal buffering.

Having an overloaded Status for returning errors is actually good. Saves you a lot of guessing.

the Env functionality lets the end-user override a lot of codepaths (especially with regard to filesystem operations) - this allows for some really cool customizations and feature injections. For example, one might wish to print something/log something every time a file is deleted. This is also great for wrangling cross-platform file management quirks and implementations.

see: https://github.com/google/leveldb/blob/a6b3a2012e9c598258a295aef74d88b796c47a2b/include/leveldb/env.h#L51

LevelDB also implements writer throttling - In LSM Trees, the number of files present in L0 are a dominating factor in read latency. In the tiered compaction strategy, we can enforce the invariant that due to non-overlapping SSTable ranges in every level, for every level except level 0, we can guarantee that only one SSTable file has the key we're concerned about, so the IO load scales proportionally with the number of levels. This doesn't apply to L0, where all the memtables are flushed directly in potentially overlapping ranges, so we may need to read more than one file on every read.

When the number of files in l0 go above a certain threshold, we throttle writes by trying to compact, or delay, within the writer.

see: https://github.com/google/leveldb/blob/4a0c572440c7df2f56a6f5fb5aec9e366d522edb/db/db_impl.cc#L1331

Manual reference counting - the memtable (at least; i've seen it in other places too, namely, the Version ) in LevelDB re-implements std::shared_ptr<T> semantics with manual reference counting. This is very curious and interesting to see - the git blame (possibly from a refactoring /sync commit) comes from 2011, which is also exactly when std::shared_ptr<T> was introduced as a part of C++11. Internal methods manually increase the reference count of this memtable.

The only scenario I can think of is that if some iterator is reading from the memtable and for whatever reason the memtable is flushed, we don't want the memtable to disappear.

class MemTable {  
 public:  
  // MemTables are reference counted.  The initial reference count  
  // is zero and the caller must call Ref() at least once.  explicit MemTable(const InternalKeyComparator& comparator);  
  ...
  
  // Increase reference count.  
  void Ref() { ++refs_; }  
  
  // Drop reference count.  Delete if no more references exist.  
  void Unref() {  
    --refs_;  
    assert(refs_ >= 0);  
    if (refs_ <= 0) {  
      delete this;  
    }  
  }
  ...
}

the Read codepath also can cause a compaction - it looks like whenever the snapshot is edited due to a compaction, it computes a recommendation for the next level to compact. The read codepath then tries to schedule a compaction (if recommended - measured via Version::compaction_score_) in the DBImpl::MaybeScheduleCompaction codepath.

https://github.com/google/leveldb/blob/4a0c572440c7df2f56a6f5fb5aec9e366d522edb/db/db_impl.cc#L668

Growing up, I was far from an athletic kid; The fact that I was pretty chubby didn't help my case either, as the difference made itself felt during sports. I tried taking part in many, many different sports (Basketball, Football, Badminton, Cricket, even) but none of them really clicked until I started running. I distinctly remember being the last one to finish in the 400m/800m running trials in school multiple times over. Outside of that, I absolutely hated running; It made absolutely no sense to me that people would enjoy the constant torture of being out of breath and struggling to move even an inch more without wanting to lie down. Fast forward five years, and I've accomplished what I think is a pretty neat thing. How did we get here?

Though I'd started with the "Couch to 5k" program in December 2024, I bought my first pair of running shoes in Jan 2025 and was keeping to a very unstructured, c25k-ish program just based on vibes. Around February of 2025, I experienced a loved one having a pretty severe health scare; this was an inflection point for me, because I realised that if I didn't change how I was living my life, I would be next. Cardiovascular fitness became a top priority for me. I took the plunge and bought a Garmin Forerunner 265, and decided to take my running more seriously.

Discovering Interval running (more along the style of the Run-Walk-Run method) was a life-changer for me. It suddenly made it possible for me to consider that I could do more than a 2km long walk (my longest yet) - and soon I was doing 3km in intervals, running for three minutes, walking for 30 seconds, and so on until I hit the mark. I slowly pushed that to 5km, and before I knew it, in April 2025, I ran my 5K PR time at the IIITB Miles 4 Meals race. This was my first organized race, and I chose to ran it in intervals. The elevation changes along the electronic city course were a sure surprise; I never expected to struggle so much. I ended up finishing that race in around 37 Minutes. Not a fantastic time by any means (I see you, sub-30 and sub-20 folks), but it was my first race. I had voluntarily signed up to do sports, paid money, even, to do sports, and not only did I do it, I did it a little quicker than I thought I could! This was insane. It set forth a chain of events that would eventually lead to me signing up for the Wipro Bengaluru Marathon, setting myself an ambitious goal of a sub 1:10 10k.

Running

Distance running is an endurance sport. Why is it an endurance sport? Because to do it, you have to endure your mind screaming at you for half an hour / one hour / etc and be in a state of consistent physical exertion

What I learnt

showing up matters. showing up is where it starts. This is the chief learning I've had from this entire experience. It does not matter how bad your day went. It does not matter if you're tired and want to lie down. If your workout for the day is a 7:00min/km 3k interval workout, and all you can manage is a 13:00min/km 1k, or lesser, or if you go a few steps and turn back, that's still a win. You got out of whatever you were doing, got ready, stretched, whatever you had to do, and you started. That's the win.

The converse is also true; some workouts you think will be really hard, or that you'll end up being unable to do, turn out to be not so hard or not as bad, simply because you didn't let the fear/anticipation of the workout change how you showed up. You prepared yourself, laced your shoes, and presented yourself to the challenge, ready to take it on. This is the win.

You can do so much more than you think you can - If you had told me five years ago that I walked (let alone ran) 10km, I'd have laughed - but I did do it! I ended up beating my adjusted expectation of a 1:20 finish by about 3 minutes, only because I put in the work, week after week, workout after workout, stretching myself a little more each and every time. You cannot start running 10km overnight; first you run 3 (two weeks in a row), 3.5 (two weeks in a row), 4 (two weeks in a row)... all the way until you reach 10km, multiple weeks in a row. gradual progressive overload is key.

Personally, I'm rather unimpressed by people who manage to pull off impressive race times without training. If it was just about the race, then you wouldn't have to worry about methodology, sustainability, anything of that sort - the people who followed a training plan, showed up day after day, week after week, are the ones who impress me. It takes grit, and sometimes, you acquire said grit, by following a training course.

easy workouts are extremely important - my longest run of the week is a zone 2 run. I've stopped measuring my long run by pace and instead measure by heart rate instead. This has made a massive difference.

Hello! You're probably here through a link in a lab manual. This post is WIP, but I hope I've managed to motivate the need for this algorithm, as well as providing you with a few questions to think about what's really going on under the hood.

You'll find the original research paper for the Raft Consensus Algorithm here - and the website lists a bunch of amazing resources you can refer to as well.

We know that Raft is a Distributed Consensus Algorithm, and while that's right, it's a way of looking at Raft from a "What does it do" perspective, not from a "what problem does it solve" perspective. Keep reading - I'll briefly motivate the problem, and you'll see why this algorithm is necessary.

Motivating the problem

Let's consider that you're the lead backend developer for a hot new startup, fonbuk, which provides people with an online phonebook.

The storage backend for your application is a python dictionary, quite literally this -

{
  "user1": {
    "ABC": 123567890,
    "DEF": 123567890,
    "GHI": 123567890,
  },
  "user2": {
    "ABC": 123567890,
    "DEF": 123567890,
    "GHI": 123567890,
  },
  ...
  # And so on
}

The operations on this look as follows - any user can only set and get phone numbers, and these are the interfaces that any sort of clients will use.

def set(phonebook, user_name: str, contact_name: str, number: int):
  ...

def get(phonebook, user_name: str, contact_name: str):
  ...

We'll walk through the stages of growth that your company goes through, and the journey you face as the developer.

Stage 0: Single User, Single Machine

• Scale: 10 users per day

In this part, you're the only user - you get to handle all the queries to read and write numbers, since you're an early-stage startup - so there's nobody else to do this. Your scale, 10 users per day, means you can very comfortably manage to run this on a single machine.

Since you're the only user, you get to decide what order you save phone numbers in - if two people want to change the same contact at once, you decide whose write "wins". This is important.

Stage 1: Multiple User, Single Machine

• Scale: 100 users per day

Let's say you manage to make it to TechCrunch Disrupt, displacing title favorites "Pied Piper" for the prize - your innovation has won hearts! People start running toward your app for their phone needs. At this point, you need to handle multiple users!

So, small problem. While your single-user, single-computer model worked fine, you've now got multiple people reading from and writing to the same piece of data. This introduces a bunch of problems, such as -

1. If two people write to the same sub-item at the same time, whose write is correct?
2. If Person A did set(pb, "anirudh", "somesh", 123) and then person B did set(pb, "anirudh", "somesh", 123) but Person B's write reached the server before Person A's did, whose write should be considered?

The answer to this is that for "consistency" (i.e. having the "right" value in place according to some scheme), it's necessary for someone to mediate these conflicts. Since you're a capable developer, you implement this - but soon enough, you begin to see some problems.

Stage 1.5: Single User, Multiple Machines

• Scale: 1000 Read-Heavy users per day (reads outnumber writes in a 100:1 ratio)

You decide to carry out an experiment. Computers have limits - there's only so many requests one computer can handle, so you decide - since most people on this application are reading phone numbers rather than writing, I can have multiple computers to handle the read requests.

So, There's only one user who gets to decide the order of things (that's you) and who gets to mediate all conflicts - that's also you.

However, here's a small problem - if you update a value by using set on one computer, how do all the other computers know this value has been updated? Think about it as cache invalidation of a sort. Since there's only one user, you can do what's known as replication to let all the other computers know that something's been changed here, and what it's been changed to.

The challenge here is maintaining consistency between the different copies of state on various machines so that they can handle all these queries independently. Since there's only one person deciding what's being written (you), there are no problems.

Stage 2: Multiple Users, Multiple Machines

Scale: 100,000 Users

Congratulations on hitting scale! Your startup now needs the best, fastest data solution you have without burning a hole in their pockets - And you come to the rescue! Since you want to take advantage of parallelism, you have multiple machines, and since you don't want to make any user (or human) being the bottleneck, you let people access these machines in parallel.

This brings you to a host of interesting problems, such as -

1. There's no centralized conflict resolution - multiple computers may see multiple values as the correct value at any given point in time, leading to inconsistency
2. There's no easy way to tell whether or not a computer has the latest value
3. If you write a value, there's no guarantee that all computers have recieved that write - maybe some overwrote it!

The Solution

A bunch of really smart people a while back figured out that a good and acceptable (not perfect) solution was to get the computers to agree on whether or not a value should be accepted by everyone as the truth.

This works, because

1. All Computers agree on the ordering of writes, so there are no conflicts - taking care of our conflict resolution problem
2. All Computers have the same state post-agreement, taking care of our replication problem
3. All Computers will only give back the latest value it has, and because all computers agree on any new values, this will be the latest value overall.

Awesome! We now understand what Distributed Consensus is - it's when a bunch of computers agree on a certain decision that should be taken, which all other computers agree to.

What's Raft, then?

Raft is a distributed consensus algorithm that works by separating the problem of distributed consensus into three distinct parts (Leader Election, Log Replication, Safety), and solving each on its own.

The TL;DR Black Box overview is this - Any group of computers using Raft for consensus on any decision will be able to guarantee (within reasonable bounds) that all computers have the same decisions in the same order.

To put it more formally, we can think of Raft as an algorithm to manage a replicated log. If we look at our Phonebook example, it turns out that we can represent operations on that phonebook as a series of log entries.

SET ANIRUDH ABC 123
SET SOMESH DEF 134

The Log above, when applied in the same order, gives you ->

{
  "ANIRUDH": {
    "ABC": 123,
  },
  "SOMESH": {
    "DEF": 134,
  },
}

The Key realization is that the log is a series of actions, and when applied to any FSM, you'll end up in the same state for a deterministic FSM - so if you manage to "replicate" the log of actions keeping the order intect, technically... you've managed to replicate the state! Even then, that has some limitations, but we won't go there today.

Not to mention the fact that you can represent a lot of data structures like this, including databases :)

How does Raft Work?

Raft Node States: https://www.eecs.berkeley.edu/~rcs/research/raft_fsm.png

Let's assume a cluster of five nodes, each of them in the Follower state by default.

Each of them is equipped with a randomized timer, ticking down from the time the node comes alive. Eventually, someone's timer expires, and when that timer expires, the node (say node 1) stands for election by moving to a candidate state.

All the other nodes in the cluster look at this and go "hey, someone who wants to do my work for me :D let's vote them in", thus, appointing a "term leader" for the current "Term" of the cluster. A term is just a logical clock, and you can think of it as a number that increases every time a new leader is elected.

So, once this leader is voted in a by a majority of the servers, it now has full control - i.e. all writes that all followers need to do get forwarded to the leader, whereas the followers can serve read queries. Remember - All Writes Flow to the Leader! There can only be one leader per term.

Once the leader recieves a write and has made sure that a majority of nodes in the cluster have recieved that write, it can then "commit" that write, which means it's safe for all other machines in the cluster to assume that the value will not be lost in the event the leader crashes.

What happens if the leader crashes?

The leader checks the status of each node in the cluster every once in a while. Every time a node hears from the leader, its timer gets reset.

Simple! Another node whose ranom timer expired will stand for election, become the next term leader, and will then guide the rest of the nodes on what to do. This is why Raft is Fault-Tolerant.

What's the decomposition?

1. Log Replication - Leaders have rules on when they can and can't tell nodes to write a value. Nodes, too, will perform a bunch of checks - to see if they request to write is one that's consistent with the previous history of the node.
2. Leader Election - Raft's use of timers makes it particularly safe from split-brain problems, such as two leaders in the same cluster - the timers also allow for deterministic randomization of the leader election process.
3. Safety - Nodes will only vote a candidate into leadership if the candidate has the latest data. This is determined using term numbers.

This decomposition not only makes the algorithm easier to understand but also easier to build.

Overarching WIP Conclusion

You definitely should learn Raft. This visualization is an excellent one - try doing all sorts of things like stopping nodes, adding new ones, and so on.

We start clean.

Hello! Let's understand the Raft Consensus Protocol.

You might be asking - What the hell is Raft? And you're right. there's no clear answer. Is it a boat? is it a plane? Is it Batman? And these are all valid questions. There's only one right answer, which is that Raft is a distributed Consensus Algorithm.

Sometimes just one computer isn't enough, ya know? Sometimes you just need more juice. Sometimes your one dusty old AMD Athlon Dual Core from 2006 can't keep up with the load, which means you need to scale.

Scaling, our benevolent dictator, manifests for computers in two ways - Vertical Scaling and Horizontal Scaling.

• Vertical Scaling Increasing the amount of CPU, RAM, Disk Space available on each machine - as you go higher, this becomes prohibitively expensive
• Horizontal Scaling Get a bunch of lower-power machines and make them process the data in parallel! It's automatically parallel without switching overhead et al and is also cheaper.

Since vertical scaling has its limits and is expensive, horizontal scaling is often the recommended option. You'll see some notable holdouts to this - such as Stackoverflow - but I digress.

Ultimately, if you want multiple machines coordinating on some single task, you'll need them to talk to each other to figure out what to do and how to do it! In some cases you usually have someone coordinating these things (a la Zookeeper, HDFS, etc) or you have these decisions being made autonomously amongst these machines themselves, without you having to appoint a "leader".

Getting machines separated by a network, all working towards the same thing (i.e. a Distributed System) to agree on a descision (one thing) is known as a consensus algorithm.

Mutex unlocks are implicit in rust - this means that you cannot explicitly unlock a mutex in Rust, and that the Mutex<T> does not implement an unlock method in Rust. This means that you need to be really careful about how you lock and unlock Mutexes in Rust.

Consider the usecase of a threadpool - threadpools use parallelism to increase throughput in a request-based system like a web server. This allows the server to work on multiple requests in Parallel. So this usually requires incoming requests to be placed into a queue for workers to consume from.

In a multi-threaded system, each worker thread must occasionally check this queue to see if there are jobs on the queue, and must pick one job off the queue to handle. This allows multiple threads to work on jobs in parallel, reasonably(include amdahl's law ref) increasing throughput of the system.

To do this, we use a Mutex lock on the queue, to prevent that only one worker can remove one job from the queue at any given time. We do not care which worker thread it is, so long as there is only one. If we do not do this, we can have messy situations like two worker threads picking up the same job and working on it, leading to a host of problems like overwritten responses, processing it twice overall, etc -> so the worker thread's job is basically

• Check the queue - see if there are any jobs for me
• Pick the first available job for me
• process the job

At times like this, the question of "how long does a worker thread hold the lock on the queue" is of crucial importance to the throughput of the system. Ideally, you want to unlock as soon as you finish reading the task (usually a function pointer or the task description) from the queue. Especially in a web server, since it isn't easy to predict how long each request will take, it's better to unlock as soon as you consume the job, and not hold the lock until the job is done executing. This allows other threads to get their jobs faster.

Funny thing is in Rust there's no way to explicitly unlock a mutex - it automatically unlocks when the lock falls out of scope. Now you might say -

That's stupid!

And I'll agree with you - but idiomatic rust will show up and smack us both on the head. Idiomatic Rust basically encourages this to happen implicitly.

// This causes contention! `held_lock` is not going to be left alone until the task is
// finished.
let held_lock = channel.lock().unwrap();
let current_task = held_lock.recv().unwrap();

In this case, the lock denoted by held_lock is going to remain held until it reaches the end of the current scope. Assuming you'll be executing your task after this, it's bad news for the performance.

But.. what if you could somehow fool the lock into thinking it's reached the end of the scope? Rust also allows you to do this by introducing arbitrary scoped blocks wherever you want. So we just put the lock into the scoped block, and the mutex is released once the block is dropped!

// This does not cause contention! You're using a custom scope block to drop the
// MutexGuard, hence unlocking the channel consumer, and letting other threads get jobs
let current_task: Job = {
	let held_lock = channel.lock().unwrap();
	held_lock.recv().unwrap()
};

Without proper error handling, you're more likely to do this when you're prototyping. If you haven't read this then you're also likely to put it in prod.

// This is the same as the above - because the MutexGuard that's returned after the
// first unwrap is dropped after the recv method
let current_task = channel.lock().unwrap().recv().unwrap();

idk why this decision was taken. It appears to be something of minor controversy, but as of right now the correct way (sans .unwrap) appears to be the scoped block option.

You can now finally stop thinking about your code as a pencil sketch, and you can now finally start painting. You can afford to now pay attention to the finer details - what is this function supposed to be called? Do we really need all the data in this struct? In other words, you can now start refactoring.

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” ~ Martin Fowler

A well-timed refactor can save you a lot of time and energy; we've all been in the situation where we've looked at code we've written a while back, and have understood none of it. Context - the reason you can get away with giving variables single-letter names - fades, and leaves behind a massive void. By refactoring your code for maintainability, you're looking out for your future self!

Now that you've been convinced that this is important, let's walk through an example. The goal here is as follows -

Refactor a large function into multiple smaller functions with minimal breakage

The Problem

Consider the following function, built by our friendly neighbourhood widget factory elves. As you can see, they're doing their best, but in the interest of iteration, they've ended up with a massive, massive function.

// main.go
package main

import "fmt"

func FizzBuzzOverkill(limit int) ([]string, error) {
	var final_output []string
	for i := 0; i < limit; i++ {
		output := ""
		div_three := false
		div_five := false
		// check for divisibility of three
		if i%3 == 0 {
			div_three = true
			output += "fizz"
		}
		if i%5 == 0 {
			div_five = true
			output += "buzz"
		}
		if !div_three && !div_five {
			output += fmt.Sprintf("%d", i)
		}
		final_output = append(final_output, output)
	}
	return final_output, nil
}

The Solution

How messy! Certainly, we can do better than this - but refactoring without tests, atleast end-to-end tests, is a bad idea! you need some baseline for correctness.

// main_test.go
package main

import (
	"testing"
	"golang.org/x/exp/slices"
)

func TestFizzBuzz(t *testing.T) {
	sample_output := []string{"fizzbuzz", "1", "2", "fizz"}
	generated_output, err := FizzBuzzOverkill(4)
	if err != nil {
		t.Fatal(err)
	}
	if !slices.Equal(sample_output, generated_output) {
		t.Errorf("Fizzbuzz Failed!")
	}
}

There we go! Given that this test passes, we can now start to begin refactoring.

The guiding idea here is that you do not attempt to rewrite it all at once; you isoldate the codebase into blocks, convert each function into a block one by one, and then slowly repalce blocks with their respective code. At the end of each integration, run the test to see if it's working well!

Let's see how we do that. In the example here, we can see three major tasks within the loop:

1. Check for divisibility by 3, and
2. Check for divisibility by 5, and
3. Add the number itself if it isn't divisible by either

Given that we've identified the major working units of the code, let's begin by pulling out these parts.

An important point to note is these refactors work better, and more specifically, this method, works better when you majorly deal with pure functions. We don't want any ghosts in the shell!

// Function to add "fizz" to the string if it's divisible by 3
func ModifyStringIfDivisibleBy3(number int, output_string *string) {
	if number%3 == 0 {
		*output_string += "fizz"
	}
}

This is slightly convoluted, but you get the idea. Let's now integrate it into the main codebase.

// main.go
func FizzBuzzOverkill(limit int) ([]string, error) {
	var final_output []string
	for i := 0; i < limit; i++ {

		output := ""
		div_three := false
		div_five := false

		// check for divisibility of three, set div_three to true if it works
		div_three = ModifyStringIfDivisibleBy3(i, &output)

		if i%5 == 0 {
			div_five = true
			output += "buzz"
		}

		if !div_three && !div_five {
			output += fmt.Sprintf("%d", i)
		}

		final_output = append(final_output, output)
	}
	return final_output, nil
}

running our tests -

anirudh@shatterdome:~/projects/fizzbuzz-overkill  $ go test .
ok      fbo     0.001s

We are now good to go! We can now shamelessly refactor. Let's change, piecewise, the other two segments of the program.

Let's remove the divisibility check for five -

// Function to add "fizz" to the string if it's divisible by 3
func ModifyStringIfDivisibleBy3(number int, output_string *string) bool {
	if number%3 == 0 {
		*output_string += "fizz"
		return true
	}
	return false
}

// Function to add "fizz" to the string if it's divisible by 3
func ModifyStringIfDivisibleBy5(number int, output_string *string) bool {
	if number%5 == 0 {
		*output_string += "buzz"
		return true
	}
	return false
}

func FizzBuzzOverkill(limit int) ([]string, error) {

	var final_output []string

	for i := 0; i < limit; i++ {
		output := ""
		div_three := false
		div_five := false

		// check for divisibility of three
		div_three = ModifyStringIfDivisibleBy3(i, &output)
		div_five = ModifyStringIfDivisibleBy5(i, &output)

		if !div_three && !div_five {
			output += fmt.Sprintf("%d", i)
		}
		final_output = append(final_output, output)
	}
	return final_output, nil
}

anirudh@shatterdome:~/projects/fizzbuzz-overkill  $ go test .
ok      fbo     0.002s

And now, lastly, the catch-all.

At this point, it's a slightly controversial take - I think it's okay to leave it as is at this point. This may be biased by this particular example, but i'm trying to promote responsible refactoring and I think you shouldn't do anything more to this. It's okay.

Eliminating the redundant variables, we get:

func FizzBuzzOverkill(limit int) ([]string, error) {
	var final_output []string

	for i := 0; i < limit; i++ {
		output := ""

		div_three := ModifyStringIfDivisibleBy3(i, &output)
		div_five := ModifyStringIfDivisibleBy5(i, &output)
		if !div_three && !div_five {
			output += fmt.Sprintf("%d", i)
		}

		final_output = append(final_output, output)
	}
	return final_output, nil
}
// The line breaks added are purely for aesthetic reasons!

Why this?

The advantage of this approach is that you get to keep your refactoring process alive, while also ensuring your code works. It seems somewhat rudimentary now, but it was definitely helpful to me when I had to refactor a large function.

Happy Refactoring!

\m/

Summary: Error Handling in Rust

• Panics are bad!
• Rust enables us to use errors as variables and types as opposed to errors as exceptions. This is better for composability.
• using .unwrap() is almost always not a good idea, errors can be handled so much better -> this is because .unwrap() forces a panic!().
• It's good to know what Option<T> and Result<T, E> look like internally as this knowledge allows us to handle errors so much better.
• It's also good to know that Option<T> == Result<T, ()>, and that you can convert between the two
• If you're using .unwrap() multiple times in a row, it's probably a better idea to use a special combinator type to allow composition.
  • These combinators (.and_then(), .unwrap_or(), .or_else(), .map()) allow us to elide repetitive stuff from the code and make the code easier to read.
• Combinators are sometimes hard to read and reason about (especially if you don't know the underlying implementation) -> this is why it's usually more readable (in my opinion) to stick to implicit case analysis by using things like the try!() macro and the ? operator in conjunction with early returns
• If you're combining error types, you can explore multiple options, including, but not limited to -
  • Converting every error to a String - you lose contextual information like io::ErrorKind
  • Using Box<dyn Err [+ Send + Sync]> - same problem - because we're treating the error here as a Trait Object, we lose all other information about it.
• The best option for library authors is to define their custom error types (see stdlib definitions of errors as a template), which also define std::from::From<T> on the error types that they depend on
  • this is so that you can easily use the ? operator to handle unwraps and result err matches
  • Any other error type will try to be converted to your custom return error type, so it's good to define the std::from::From<E: Err> trait on your custom error type.
• Lastly, levels of error handling ordered by difficulty of understandability
  • TODO: Code Examples
  • Combinators
  • Explicit Case Analysis
  • Inline implicit unwraps but top-level catch

in DBImpl::Write

• create a Writer object that holds the write batch
• lock the mutex (RAII Scoped Lock) and wait for the current write to be at the front of the writer queue
  • how do multiple writers lock the same lock at the same time? or do they all contend on the MutexLock l(&mutex_)?
• if we need to throttle, then we wait - DBImpl::MakeRoomForWrite
• get the last sequence number
• the Writer now holds responsibility for locking the memtable and WAL, so the main writer thread can release the lock
• WAL commit: https://github.com/google/leveldb/blob/4a0c572440c7df2f56a6f5fb5aec9e366d522edb/db/db_impl.cc#L1236
• if WAL write was successful, write to memtable https://github.com/google/leveldb/blob/4a0c572440c7df2f56a6f5fb5aec9e366d522edb/db/db_impl.cc#L1245
• Lock the mutex again, and update the high sequence number
• remove the writers from the queue (why does this happen?)
• signal the waiting head of the write queue

Read Path

in DBImpl::Get

• Create a scoped RAII Lock for the mutex
• Get the snapshot (as determined by a Sequence Number) for the read (either via the ReadOptions or by getting the latest Sequence Number)
• Create a LookupKey
• refcount the latest memtable, immutable memtable, and Version
  • what is Version? Version appears to be how LevelDB represents an on-disk snapshot. Conversely, VersionSet is what appears to represent a "snapshot manager"
    • Each Version is a node in a linked list; it links to the next and previous version as well
    • It has a list of files in each level
  • what is the manifest_file_number_? Looks to be the snapshot ID, so that we have clear versioning amongst the snapshots.
• Unlock the mutex and lookup the key in the memtable
• If the key is not present in the memtable, look up the immutable memtable
• if the key is not present in the immutable memtable, look up the current snapshot (see Version::Get)
  • create a tracker struct for lookup information
  • call into Version::ForEachOverlapping - takes a callback to run on every file whose key range matches the lookup key
    • the callback does the actual work of opening the file and reading from it - when the callback returns false, the search stops and returns
    • first search level 0 in order of newest to oldest - process all level 0 files that overlap and run the callback on them
    • if the callback doesn't return, proceed to search lower levels (1, 2, 3, 4)
      • important to note: the guarantee we have now is that only one file will contain the information for any key - so first we search for the first overlapping file within the level, then we search within the file
      • for each level, binary search the files until we find the first file whose high key is greater than the lookup key
        • if the lookup key is smaller than the low key of the file, then we skip the file
        • the lookup key is larger than the low key of the file, we can look within the file
  • Within the callback:
    • update internal stats tracking for what the last file read in the read was
    • look up the TableCache within the VersionSet
      • TableCache is an LRU Cache (exposed as the interface Cache) that maps keys to values, and maintains some other metadata for a key in a Handle. The TableCache exists because opening an SSTable (lots of work regarding decoding the index blocks, filters, etc) is an expensive process, so we can cache the file metadata in a cache and avoid having to read it on every read for the key.
      • This opens the file (if not already open) and searches the contents of the Table in a block-wise fashion (Table::InternalGet) - if the search is rejected by the bloom filter, return early
      • Once the block is found, seek to the key
      • if the key is present (i.e key we seeked to is equal to search key, not greater/lesser), and not corrupt, and not a deletion, then capture the value
        • and return false
    • notes: seeing a lot of "charge against/for", wonder what this means (could this be the usage count?)

LookupKey

This is a special class used to make lookup operations easier in LevelDB. When the user performs a lookup with DB::Get(), the key as the end user sees it - a Slice - is converted into a LookupKey. This serves multiple purposes - you have only one type of key that you use for every operation here on out, whether it's the memtable lookup or the lookup from the files on disk.

An interesting thing to note is the small key optimization, which allocates the key on the stack within the LookupKey class itself if the key is smaller than 200 characters. This prevents a trip to the allocator and the memory subsystem, which generally protects performance at the cost of slightly higher memory usage - but given that the LookupKey is relatively ephemeral and the allocation is fast (i.e. stack frame allocation speed), this temporary wastage is an okay tradeoff to make. It's no doubt that this exact number is based on a combination of heuristics and statistics from the experience of running levelDB in prod for many workloads.

We also see that the SequenceNumber (that we now know to denote a snapshot boundary) is encoded within the byte stream of the key. This is a helpful way of tracking the snapshot we're using for lookup, and, potentially, filtering out results that exist past that snapshot boundary.

https://github.com/google/leveldb/blob/863f185970eff21e826e5fe1164a6215a515c23b/db/dbformat.h#L183

// A helper class useful for DBImpl::Get()  
class LookupKey {  
 public:  
  // Initialize *this for looking up user_key at a snapshot with  
  // the specified sequence number.  
  LookupKey(const Slice& user_key, SequenceNumber sequence);  
  
  LookupKey(const LookupKey&) = delete;  
  LookupKey& operator=(const LookupKey&) = delete;  
  
  ~LookupKey();  
  
  // Return a key suitable for lookup in a MemTable.  
  Slice memtable_key() const { return Slice(start_, end_ - start_); }  
  
  // Return an internal key (suitable for passing to an internal iterator)  
  Slice internal_key() const { return Slice(kstart_, end_ - kstart_); }  
  
  // Return the user key  
  Slice user_key() const { return Slice(kstart_, end_ - kstart_ - 8); }  
  
 private:  
  // We construct a char array of the form:  
  //    klength  varint32               <-- start_  
  //    userkey  char[klength]          <-- kstart_  
  //    tag      uint64  
  //                                    <-- end_  
  // The array is a suitable MemTable key.  
  // The suffix starting with "userkey" can be used as an InternalKey.  
  const char* start_;  
  const char* kstart_;  
  const char* end_;  
  char space_[200];  // Avoid allocation for short keys  
};

Looking at the constructor gives us a little more of a hint as to what's going on, and how exactly the sequence number is encoded.

LookupKey::LookupKey(const Slice& user_key, SequenceNumber s) {
  size_t usize = user_key.size();
  size_t needed = usize + 13;  // A conservative estimate
  char* dst;
  if (needed <= sizeof(space_)) {
    dst = space_;
  } else {
    dst = new char[needed];
  }
  start_ = dst;
  dst = EncodeVarint32(dst, usize + 8);
  kstart_ = dst;
  std::memcpy(dst, user_key.data(), usize);
  dst += usize;
  EncodeFixed64(dst, PackSequenceAndType(s, kValueTypeForSeek));
  dst += 8;
  end_ = dst;
}

Over here we see that when LevelDB is deciding whether or not it needs to allocate more space for the key to be stored overall, they compute size_t needed = usize + 13 - possibly accounting 5 bytes for the Varint32 and 8 bytes for the Fixed64 that the key size and the sequence number are stored as, respectively, but I was confused as to where the extra 1 is coming from (i.e. why we're using 5 bytes to store a 4-byte integer). Looking at other functions for Varint32, we see that even in PutVarint32() the authors use 5 bytes as a default storage length, confirming that this isn't an approximation of some sort. Let's see why this number is 5.

void PutVarint32(std::string* dst, uint32_t v) {  
  char buf[5];  
  char* ptr = EncodeVarint32(buf, v);  
  dst->append(buf, ptr - buf);  
}

A small note on varints later - https://samliu.github.io/2016/10/15/varints.html I see that one bit in every byte, the most significant bit is used to indicate whether or not there's another byte coming. While this is a part of the encoding format, this is not the number itself, so the space it takes doesn't count to the value of the number. The contract that the Varint32 maintains is that you can store a 32 bit unsigned integer i.e. 0 to 2^32 - 1 . The effective capacity of 4 bytes of Varint32 is - considering only the bits available for encoding - 7 + 7 + 7 + 7 = 28, which effectively lets us store 0 to 2^28 - 1, which prevents us from storing numbers in the range 2^28 to 2^32 - 1. This is why we need the extra byte, illustrating that there's no such thing as a free lunch - adopting varints as the sole representation sometimes means that you need to use 5 bytes to store a 4-bit unsigned integer :D

My personal preference is to minimize the use of magic numbers as much as possible - for someone who's reading the code at a glance, it's much harder to understand why there's an 8 or a 13 in the places that it exists. I would create an inline const variable with a comment on top of it explaining why it exists.

Another interesting thing to note is that the encoded length of the key has 8 added to it. I couldn't figure out why this happens.

Now what we know how the encoding works, we can safely summarize that the internal_key_ is simply the key the user passed in for the lookup, with the sequence number appended to it.

"if you want to find your purpose in life, think of what you did as a child that made the hours feel like seconds"

and that got me thinking; what made me feel like that as a child?

What comes to mind immediately is Mechanix. Mechanix was a "build it yourself" toolbox, which gave you parts, and an instruction manual that showed you how to use those parts to build, step by step, a lot of things: a car, a bike, a helicopter, etc.

What I'm now thinking about is: though I really like software engineering, I'd love to bring about this feeling, the sheer joy of play, into my field. How do I do that? Some particular things stand out to me.

1. Having a clear end goal of what the finished product looks like: this gave an immediate justification to why I was doing what I was doing

2. Knowing what the parts are and how the parts fit in with each other: this was really important because once you had decided to build a smaller sub-component of the project, you had to know what parts to use, and you would have to know how the smallest most atomic parts would be combined. Where would the screws go? Could you position it in that particular way? What about the other way?

3. Using your tools well: almost every join required that you use a screw and a bolt to hold the parts together. Though it would have sufficed to know how to just "make it happen", for some joins you had to hold the spanner in one particular way, for the other you needed to know which hand to use for the screwdriver and which hand for the spanner for the easiest way to put that screw in place without messing things up

4. Being able to test the joins: once you put a screw and bolt in place, it was super important to jiggle the parts a little to see if the join was done correctly. i remember this being an important part of making sure that the thing you build wouldn't just fall apart.

5. The joy of seeing it finally working: once it was done, being able to see all the parts you put in place work in harmony to make the final build work was awesome. Building it and showing it off was a viscerally happy part of the entire process.

6. Cleanup: working on a sheet on the floor or on a table meant that I was able to easily spot screws that I've dropped and could easily find the stuff I put down.

Hello, I'm Anirudh!

• Backend and Distributed Systems Dev
• Databases Enthusiast (I call feb 14th WALentine's day)
• 3.99999th Year CSE @ PES
• Lab Head, PES Innovation Lab
• Rust, Golang
• accidental poet

What is Saaru?

• Saaru is a Static Site Generator -> In simple words, it takes in Markdown files, a bunch of templates, and spits out a whole website.

• Why is it called Saaru?

  • Saaru is the Kannada word for rasam and I love rasam

Architecture

                         STATIC SITE GENERATOR INTERNALS

      ┌────────────────┐
      │ Markdown Files │--------\
      └────────────────┘        |
                                v
      ┌────────────────┐     ┌─────────────┐    ┌──────────────────┐
      │ Template Rules │---->│ Static Site │--->│ Full Built HTML  │
      └────────────────┘     │ Generator   │    │ and CSS Website  │
                             └─────────────┘    └──────────────────┘
      ┌────────────────┐        ^
      │ Additional     │--------|
      │ Metadata       │
      └────────────────┘

• The content lives as .md files enriched with frontmatter, which is YAML, TOML or JSON that you can add at the top of a markdown file.

• You combine these files with a template, which can literally be a string with special blocks to substitute text in, or a more specialized dialect of templating like Jinja, Nunjucks, etc

• Saaru is built on combining a Markdown Engine, a Templating Engine, and Content Injection

                      MARKDOWN ENGINE

         ┌────────┐      ┌────────┐        ┌─────────────┐
         │.md file│----->│Markdown│------->│compiled HTML│
         └────────┘      │Engine  │        │output       │
                         └────────┘        └─────────────┘

• Saaru is a static site generator in Rust
  • (show docs folder) Give it a folder with Markdown Content and Some Templates, Saaru will render a website (show build folder)
  • Deep Data Merge: You can have content show up in indices like tags and collections, so that you can reference them in other parts of the site

• Switches from a one-pass traversal of the dir tree to a two-pass approach (or does it? :sus:)

  • The frontmatter
  • Controls the title, a bunch of other stuff - here's what the frontmatter looks like

• Frontmatter is metadata you can control. Here's what this looks like -> Each .md file has this, and this can control how it gets rendered.

#[derive(Clone, Serialize, Deserialize, Debug)]
pub struct FrontMatter {
	pub title: Option<String>,
	pub description: Option<String>,
	pub date: Option<String>,
	pub tags: Option<Vec<String>>,
	pub collections: Option<Vec<String>>,
	pub wip: Option<bool>,
	pub template: Option<String>,
	pub link: Option<String>,
	pub meta: Option<Value>,
}

Live Demo (with my website)

• Here's a working Saaru website. See the structure, and the frontmatter.

commit walkthrough from the bottom up

• Frontmatter struct represents a file, content, tags and all

talk about how initial R&D Happened

• Genesis commit - (https://github.com/anirudhRowjee/saaru/commit/974009349027a4f14a69959b8a1e87c74dcad0b4)

  • Goal: Render a single file of markdown using a template engine
  • was still using pulldown-cmark at that time
  • no focus on dynamic naming
  • include_str!() to read the file!

• Attempting to remove the frontmatter myself

  • learnt about .map() and .fold() used together (thanks Kiran!)
  • Added a minimal test suite
  • Led me to think "damn, i really need to reorganize this codebase" (https://github.com/anirudhRowjee/saaru/commit/bd03832a571c77cd32a2ec14627f94c42a75141f)

• Load the template name dynamically based on frontmatter

  • Learnt more about minijina environments and how they can pick up templates automatically based on directory
  • coming up with ONE function to render a single markdown file based on its templates

• How easy it was to move from the system-native directory walker (fs::read-dir to the WalkDir::new walker (which is recursive)

  • Most satisfying refactor of my life -> single-line change and required nothing else to be touched
  • https://github.com/anirudhRowjee/saaru/commit/c1553cad881013eef1d8aa2151f9488265ffaec4

• Early Deep Data Merge (https://github.com/anirudhRowjee/saaru/commit/b2185239455ec3386ba2f503409281a0ad066f69)

  • Conversion from a one-pass operation to a two-pass operation (introducing alternate_render_pipeline())
  • DO NOT WALK THE FS TWICE -> INDIRECT TO tag_map, collection_map, frontmatter_map
  • Read the FS into these maps, and iterate over frontmatter_map to render
  • These maps can pull from tag_map and collection_map to render more content based on tags and collections
  • Added proper structured logging here

• Deep Data merge (https://github.com/anirudhRowjee/saaru/commit/badcac9b06f44731c3a8cac5abf00cdf21914f10)

  • More refactoring! refactor all the things!
  • moved most stuff from src/main.rs to src/saaru.rs
  • premanently switched to two-pass render pipeline (instance.recursively_render_from_directory() -> instance.alternate_render_pipeline())
  • Load data into the tag_map, pass it to template rendering context
  • move to thin_augmented_frontmatter -> frontmatter without the raw markdown content
  • Lastly added collections (alternate ontology to tags) and a system-generated default tags page

• markdown engine switch

  • Added Command-line Arguments (https://github.com/anirudhRowjee/saaru/commit/feab2d7f2224bcd920ef95d1bd774c5c18b01cf6)

• Switched from pulldown-cmark to comrak (https://github.com/anirudhRowjee/saaru/commit/2d60a103e72e6e38d5460ba77ef1aeca212a9c1f)

  • Why?
    • more features -> Footnotes, etc
    • Better compatibility with common requests to deviate from GFM Spec
    • more configurability and much simpler API, although less suitable for user-injected AST-level customizations
    • Automatically ignores frontmatter!

adding other features

• Static folder copying for CSS, etc (src/utils.rs -> pub fn copy_recursively, shamelessly stolen from the internet)
  • Basic JSON content injection
  • Skipping non-markdown files in the parse tree
  • Yet Another Refactor!

live reload

• Added basic live reload with notify -> loop {poll_fs, if file changes, rerender that file in-place with instance.render_individual_file(filepath)} (https://github.com/anirudhRowjee/saaru/commit/c612455d70d911897742456dd0951c8b91dbcb1c)

  • had to manually refresh browser and external web server!
  • Initially architectured it as a separate module that consumed the saaru instance
  • Re-architected when adding a live server (https://github.com/anirudhRowjee/saaru/commit/b5f96fff134ab1db656872de8e19a7acb196becb)
  • New Fancy Diagram!
  • crossbeam MPMC channels, watching for fs change, waiting to notify Saaru Instance as well as web server (for live reload) -> notify, tower, tower-http-livereload

• 2 lightweight threads (watcher thread, message passing thread)

  • Hid these configurations behind command line flags
  • Specific actions for different types of files
  • .md -> Optimized re-render of single file only
  • .css -> separate watcher thread for static files, copy that specific file over into the build folder if there's a change

future plans

• DAG solver -> more fine-grained re-rendering to check what templates etc need to be re-loaded
• maybe add template livereload live while i have time

Start now. Optimize Later. Imperfect starts can always be improved. Obsessing over a perfect plan will never take you anywhere on its own.
~ James Clear

I'm currently experimenting with a new way of working on side projects, where I give myself a month to work on it, and write a blog post about it, holding myself publicly accountable via twitter.

This experience has made me realise just how untenably high my standards are - and while this has its positives, the negative is that it often tends to stifle the act and art of building with concerns that are largely useless for learning-oriented projects.

For example, if I'm implementing a skiplist - coming from the perspective of someone who doesn't know what a skiplist is - it's much more important that I understand the core algorithms and data structures of the skiplist, than it is to carve out the perfect abstractions and object-oriented design for the nodes and the skiplist themselves.

Case in point - I ultimately decided to not have any private data members in the classes, and chose to leave out some bookkeeping, and didn't handle all my errors - but I can confidently say that this is the reason I now have a working project, and was able to learn how a skiplist works.

To do this is to sketch, like artists do. We can aggressively whittle down the idea to its very core functionality, and implement that - nothing more, nothing less. This lets us establish a foundation, a feedback loop, upon which we can then "paint in the finer details" - refactor to meet our standards - in an iterative fashion.

In doing so, we choose a project that is imperfect but exists today, as opposed to a project that does not exist yet, but is eventually consistent with some ideal of perfection.

... Kids just plunge in and build their treehouse without worrying about whether they're wasting their time, or how it compares to other treehouses. And frankly we could learn a lot from kids here. The high standards most grownups have for "real" work do not always serve us well.
... The most important phase in a project of one's own is at the beginning: when you go from thinking it might be cool to do x to actually doing x. And at that point high standards are not merely useless but positively harmful.
~ A Project of One's Own - Paul Graham

At this scale, It is much easier and cheaper to go from imperfect to perfect than it is to go from non-existent to perfect. Projects that do not exist cannot be improved or learnt from.

The goal of this post is to walk everyone through how an LSM Tree works. I'll do my best to not wander off into philosophical questions and trying to reason about "why" it is the way it is, and focus more on "so what". I promise that i'll get to the rambling at some point though :D

Table of Contents

1. What is an LSM Tree?
2. Immutability as an operating principle
3. The LSM Tree Architecture
4. The Write Path
5. The Read Path
6. Formalizing our understanding of performance
7. Why is too many files bad, and how do we fix it?
8. Compaction

What is an LSM Tree?

An LSM Tree is a disk-based data structure that's used to store key-value pairs on disk. A disk-backed KV (Key-Value) Store is a foundational building block for many applications, so doing this well is important. The main focus of a key-value store is persistence, so it's obvious that the disk, and files, are involved somewhere - we're going to see how and where.

Simply put, the job of the LSM Tree (and most other storage structures) is to obey the following interface, while ensuring that the data you've added or modified stays as it is between crashes and restarts.

LSMTree {
	fn Upsert(String key, String value)
	fn Delete(String key)
	fn Get(String key) -> Option<String>
}

That's the functional requirement. Non-functional requirements are varied and many, including, but not limited to:

1. Take as less disk space as possible
2. Make the reads fast
3. Make the writes fast
4. Make everything fast (Jury's out on whether or not this is a functional requirement, but for the sake of this post, let's assume it isn't)

Compared to other, more conventional data structures that also do the same thing (for example, B-Trees), the LSM Tree really shines when used with write-heavy workloads. An important distinction is that LSM Trees do not update data in-place - data, once written to disk, is guaranteed to be immutable, and will only ever be rewritten into a new file at some point. This lets LSM Trees have much cheaper writes than other data structures that perform in-place modifications to existing data files.

Let's first explore what out-of-place updates are, and then we'll investigate how the core data structure actually works.

An Immutable Database? What the hell is that?

The fundamental question remains - how do you overwrite or delete data if the on-disk data structure is immutable? The simplest answer is - to create another on-disk data structure that represents this, and ensure you read all data on disk in the order of newest to oldest.

To motivate this idea, let's take an example of a simple JSON file (data.1) where we have three key-value pairs. The filename is important as it helps us gain an understanding of the order in which the files were written.

// data.1
{
	"backstreet boys": "show me the meaning of being lonely",
	"metallica": "one",
	"charanjit singh": "kalavati",
}

Let us say that we wish update the value of the key "metallica" to "for whom the bell tolls" under the specific constraint that we cannot modify the file itself which has the key stored in it.

To show that data has been modified, if I can guarantee that data.(x+1) is always read before data.(x) where x + 1 > x, then I can "modify" the data present by writing data.(x+1) as follows:

// data.2
{
	"metallica": "for whom the bell tolls"
}

Consider our read algorithm to be as follows, essentially enforcing the principles we mentioned earlier:

read(key):
	for file in files.sort(newest_to_oldest):
		for kvpair in file:
			if kvpair.key == key:
				return (true, kvpair.value)
	return (false, "")

In this way, we can guarantee that our reader will always first see the latest value of the key, hence performing a "modification".

Now, you might ask - this works for overwrites; but how does one delete a record? The way we do this is that we frame our deletes as overwrites. If I wish to delete the key "backstreet boys", all I need to do is create a new data file:

// data.3
{
	"backstreet boys": DELETED
}

This is known as a "tombstone". Our reader is then modified to be as follows:

read(key):
	for file in files.sort(newest_to_oldest):
		for kvpair in file:
			if kvpair.key == key:
				if kvpair.value == TOMBSTONE:
					return (false, value)
				return (true, value)
	return (false, "")

Yay! We now have an immutable database. Now, you may be wondering: if all we ever do is write new files, even if data is deleted, how do we avoid making the disk full? Hold on to that question, and let's explore how an LSM Tree actually works.

What makes an LSM Tree?

A record is a pair of strings/bytes, usually both key and value. We can consider a tombstone/deletion marker to be some type of string/byte stream as well, recognized throughout the LSM Tree. We can consider the LSM Tree to be a collection of records, such that you perform CRUD operations via the key. Hence, the atomic unit of data in this LSM Tree is the record.

The LSM Tree usually has three components.

Memtable: This is the in-memory component of the LSM Tree. It's usually a fixed size, and all records end up here first, ensuring faster writes. It's usually a hashmap-type data structure that lets us access data via a key-value interface.

Write-Ahead Log (WAL): this disk-based component ensures that we don't encounter data loss if some records are only in the memtable and haven't made it to disk yet. All records are written to the WAL first, and then to the memtable, to ensure that the records have been persisted to disk - this ensures that we're able to recover memtable data if we crash without writing the memtable to disk. We won't be exploring this too much in this post, but it's important to know why it exists.

SSTable files: This is the largest component of the LSM Tree. When a memtable is full, it is "flushed" onto disk, to create one SSTable file. Each file contains a set of records stored in a sorted manner.

this isn't a component, but an important concept nonetheless - Levels: LSM Trees usually store these SSTable files in levels. There are usually five levels, and each level has a cap on the maximum size of an SSTable that can be present inside the levels. There's often a strategy to how we organize the levels themselves, and what heuristics let us keep data in one level vs move it to the other; For now, we can assume that the higher levels have smaller files, fresher data, and that the lower levels have larger files with older, staler data.

Writing some data to an LSM Tree

write(key, value)
	write to WAL
	write to memtable
	if memtable is full, write it to disk as an SSTable

Given that the memtable is in-memory, but we still need to offer durability as a guarantee, we will make use of a write-ahead log where all data is persisted to disk by default, and only once it persists in the WAL will we write it to the memtable. This ensures that if the machine crashes after the write completes to the memtable, but isn't present on disk, we do not lose our data.

Flushing Data to disk

We know that we don't have infinite memory available! Once the memtable goes beyond a defined capacity (either in terms of number of records or in terms of size), the memtable is flushed to disk, as an SSTable. This frees up the memtable to accept new writes, and ensures that our data is reliably persisted to storage.

Most implementations will usually have a second immutable memtable, which is present as a standby; this is because flushing is an operation that involves a disk IO, which, in computer time, can be pretty slow. If we block the memtable from accepting any writes during the flush, it'll be a performance penalty that we don't really want to pay - hence, flushing allows us to swap the active memtable out, replace it with an empty one (thus making the full memtable immutable), and operate on the immutable memtable.

For now, let's assume that we do nothing apart from flushing our memtable into a new SSTable. This means that every time we fill our memetable's memory quota, we will create a new file on disk. This also means that our disk size will grow in a rate proportional to the write rate.

All memtables are flushed into Level 0 of the LSM Tree, without further organization.

Reading some data from an LSM Tree

search_table(table, key)
	for record in table:
		if key == record.key:
			return found
	return not found

read(key):
	read the memtable
	if found, return
	for level in levels:
		for table in level.sort_by_recent():
			search_table(table, key)
	return "not found"

Let us first understand that to read some data, we need to know where it is. This is the majority of the problem; as data structure designers, we need to ensure that all keys are found (or if not found, confirmed as such) as quickly as possible. Once we know where the data is, reading the disk block is a relatively fixed-time operation.

To find a key, we first search both memtables. If the key was written fairly recently, we maximize our chances of a very cheap read, as we perform no disk IO for this read.

This approach does introduce some problems. Since we don't really maintain any other index, for every key we search, we need to go through every SSTable file. Relatively speaking, opening a file and reading from it is an expensive operation; We would ideally like to minimize this as much as possible. So we need to figure out some way to make reads cheaper, and to avoid opening a large number of files for each read.

To do this, an invariant we maintain is that all levels of the LSM Tree (apart from Level 0) maintain SSTables that comprise only of non-overlapping key ranges. This means that when we look at a particular level, a key can belong only to one SSTable in that level - this is to ensure that we have at most one file to search through in each level of the tree.

Once we do this, our algorithm looks something like this:

search_table(table, key)
	for record in table:
		if key == record.key:
			return found
	return not found

read(key):
	read the memtable
	if found, return
	for level in levels:
		if level == level0:
			for table in level.sort_by_recent():
				search_table(table, key)
		else:
			table = find_table() # there can only be one
			search_table(table, key)
	return "not found"

Formalizing our understanding of performance

let's look at two new terms that'll help us understand how to reason about performance in storage systems. We can use them as a sort of Big-O notation to reason about which storage algorithms are efficient, and which aren't.

read amplification: the ratio of bytes read to read a value, to the size of the value
write amplification: the ratio of bytes written to write a value, to the size of the value
space amplification: "Space-amp is the ratio of the size of the database to the size of the data in the database"¹

In a B-Tree, write amplification is high, because every write causes you to potentially split the parent node pages into new ones, which you will also have to write to disk. Conversely, the read amplification is pretty low, because you only need to read log_2(branch_factor) disk pages due to the nature of the B-Tree.

Let's go back to our case of the LSM Tree so far; we don't do anything to the SSTable files once they're written as a part of the flush process. This means that if we have a key that we want to look up, we're going to have to basically read through most, if not all, of the files present - this causes massive read amplification. This is also harmful from a performance point of view, because what could have been 1 I/O syscall with an in-memory index, is now a potentially unbounded number.

Given that the LSM Tree we know in this article so far, is optimized for low write amplification - a near perfect factor of 1, because we write all data to a new file regardless - we now need to figure out how to reduce the read amplification of the LSM Tree to make sure it's reasonably performant.

Why is too many files bad, and how can we fix it?

As of right now, we already know that we have a pretty fast write path; data hits memory first and stays in memory until the memtable is full, beyond which point we need to flush the data to disk. Measured by I/O, these writes are very, very cheap (1 I/O), but in the grand scheme of things, not free - we will now incur a cost that we need to pay on every read.

The situation we are at so far is as follows:

1. If we need to read the disk, we need to read all SSTables as they all have overlapping ranges, so we can't really say which specific table has data. It could be present in multiple tables. This will incur high read amplification.
2. Our SSTables will each be as small as the memtable size quota, which means that we'll need to be doing a large number of random disk IOs to read all the SSTables. This is the exact problem that manifests in B-Trees that we are trying to avoid (Modern day SSDs handle this much better, but we'll get to that in another post).
3. We are never reclaiming data from the disk; no matter how many times a record is overwritten or deleted, the original record still exists on disk. This is wasteful.

Given the problems, we need to implement some kind of solution that

1. Keeps the disk size from growing at an unbounded rate
2. Limits the number of small files present by grouping more data on disk together, allowing for a larger file to be created and accessed with fewer IOs
3. Maintains the invariant of non-overlapping ranges

Let's go back to the previous example we have.

// data.1
{
	"backstreet boys": "show me the meaning of being lonely",
	"metallica": "one",
	"charanjit singh": "kalavati",
}
// data.2
{
	"metallica": "for whom the bell tolls"
}
// data.3
{
	"backstreet boys": DELETED
}
// let's add some more music here
// data.4
{
	"moblack": "yamore",
	"AC/DC": "thunderstruck",
	"gorillaz": "feel good inc",
}
// data.5
{
	"charanjit singh": "raag bhairav",
	"death cab for cutie": "i'll follow you into the dark",
	"daft punk": "instant crush"
}

We can clearly see that we're overwriting some data - remember, we need only keep the latest value - and we're deleting some data too.

What if we were able to pre-compute the outcome of a read (per the algorithm we described) for this set of files? If we can ensure that the read function works the same on two different representations of data, while one takes lesser space, and is correct (in that we get the same values we stored per key, for every key), would it not be better to perform said computation in the background?

By doing this, we have solved a bunch of the problems we have, by:

1. Converting many small files into one large file
2. Removing overwritten/deleted data
3. Can also move the data into clear non-overlapping ranges

This is the process known as compaction, which is a background task that works to maintain the disk space used by frequently merging SSTable files such that disk usage is kept at a reasonable level, while lookup latency is also managed by ensuring that we don't have too many small files to open.

Compaction: Merging multiple SSTable files into one

So, let's start with the following motivation:

What do I need to do to maintain the same data, but with lesser space?

// data.1
{
	"backstreet boys": "show me the meaning of being lonely",
	"metallica": "one",
	"charanjit singh": "kalavati",
}
// data.2
{
	"metallica": "for whom the bell tolls"
}
// data.3
{
	"backstreet boys": DELETED
}
// let's add some more music here
// data.4
{
	"moblack": "yamore",
	"AC/DC": "thunderstruck",
	"gorillaz": "feel good inc",
}
// data.5
{
	"charanjit singh": "raag bhairav",
	"death cab for cutie": "i'll follow you into the dark",
	"daft punk": "instant crush"
}

A good place to start is to know that we need to maintain key ordering even in the final table, so let's start by ordering all the data (across files) by key; Since the order of writing (AKA the SSTable ID) is important, we'll keep that around, too.

[
	{"tableID": 1, "data": {"backstreet boys": "show me the meaning of being lonely"}},
	{"tableID": 1, "data": {"metallica": "one"}},
	{"tableID": 1, "data": {"charanjit singh": "kalavati"}},
	{"tableID": 2, "data": {"metallica": "for whom the bell tolls"}},
	{"tableID": 3, "data": {"backstreet boys": DELETED}},
	{"tableID": 4, "data": {"moblack": "yamore"}},
	{"tableID": 4, "data": {"AC/DC": "thunderstruck"}},
	{"tableID": 4, "data": {"gorillaz": "feel good inc"}},
	{"tableID": 5, "data": {"charanjit singh": "raag bhairav"}},
	{"tableID": 5, "data": {"death cab for cutie": "i'll follow you into the dark"}},
	{"tableID": 5, "data": {"daft punk": "instant crush"}}
]

First ordering by key:

[
	{"tableID": 4, "data": {"AC/DC": "thunderstruck"}},
	{"tableID": 1, "data": {"backstreet boys": "show me the meaning of being lonely"}},
	{"tableID": 3, "data": {"backstreet boys": DELETED}},
	{"tableID": 1, "data": {"charanjit singh": "kalavati"}},
	{"tableID": 5, "data": {"charanjit singh": "raag bhairav"}},
	{"tableID": 5, "data": {"daft punk": "instant crush"}},
	{"tableID": 5, "data": {"death cab for cutie": "i'll follow you into the dark"}},
	{"tableID": 4, "data": {"gorillaz": "feel good inc"}},
	{"tableID": 1, "data": {"metallica": "one"}},
	{"tableID": 2, "data": {"metallica": "for whom the bell tolls"}},
	{"tableID": 4, "data": {"moblack": "yamore"}},
]

Then, ordering from most recently written to least:

[
	{"tableID": 4, "data": {"AC/DC": "thunderstruck"}},
	{"tableID": 3, "data": {"backstreet boys": DELETED}},
	{"tableID": 1, "data": {"backstreet boys": "show me the meaning of being lonely"}},
	{"tableID": 5, "data": {"charanjit singh": "raag bhairav"}},
	{"tableID": 1, "data": {"charanjit singh": "kalavati"}},
	{"tableID": 5, "data": {"daft punk": "instant crush"}},
	{"tableID": 5, "data": {"death cab for cutie": "i'll follow you into the dark"}},
	{"tableID": 4, "data": {"gorillaz": "feel good inc"}},
	{"tableID": 2, "data": {"metallica": "for whom the bell tolls"}},
	{"tableID": 1, "data": {"metallica": "one"}},
	{"tableID": 4, "data": {"moblack": "yamore"}},
]

Now that we have the most recent data in a sorted fashion, we can delete all outdated and overwritten copies;

[
	{"tableID": 4, "data": {"AC/DC": "thunderstruck"}},
	{"tableID": 3, "data": {"backstreet boys": DELETED}},
	{"tableID": 5, "data": {"charanjit singh": "raag bhairav"}},
	{"tableID": 5, "data": {"daft punk": "instant crush"}},
	{"tableID": 5, "data": {"death cab for cutie": "i'll follow you into the dark"}},
	{"tableID": 4, "data": {"gorillaz": "feel good inc"}},
	{"tableID": 2, "data": {"metallica": "for whom the bell tolls"}},
	{"tableID": 4, "data": {"moblack": "yamore"}},
]

Which brings our final data sstable back to:

// data.6
{"AC/DC": "thunderstruck"},
{"backstreet boys": DELETED},
{"charanjit singh": "raag bhairav"},
{"daft punk": "instant crush"},
{"death cab for cutie": "i'll follow you into the dark"},
{"gorillaz": "feel good inc"},
{"metallica": "for whom the bell tolls"},
{"moblack": "yamore"},

We went from 11 rows and 5 files, to 8 rows and 1 file! This is the magic of compaction. Though the improvement seems small right now, this scales with your data size, and this means there are huge benefits to doing this correctly. We can now safely delete the older files, because the latest file will now answer all queries correctly in its place!

How is Compaction Implemented?

Now that we know what compaction is, let's explore how it happens in LSM Trees. This usually depends heavily on the implementation, but the general practice is that we pick a file from a level and merge it into files that have an overlapping key range in the next level. This is to maintain the invariant we spoke about in the previous part, where each level has only one file that a key can be in, which is why we need to have non-overlapping key ranges.

First, we identify the target SSTables that we need to compact. How we decide which sstables we want to compact also has massive performance implications. For now, we will just pick an overlapping key range.

Fundamentally, compaction is a merge sort on two very large sorted arrays of key-value pairs, where we maintain the hierarchy of sorting order as first by key, and then by creation time; then we simply discard all the values for a key except for the first one. This gives us an SSTable in memory!

Most LSM Trees usually use some sort of iterators to do this, as it is memory-efficient to page disk blocks in one by one, and because each table is sorted, we know that the iterator will move in only one direction. RocksDB for example has a MergingIterator² that helps them do this, with the examples provided.

Conversely, if we wish to iterate over two SSTables with a non-overlapping key range in the same level, we can simply "join" (read one after the other) the two SSTables. We accomplish this via a Join Iterator.

The final iterator construction looks something like this:

I won't cover the implementation details of the iterators now, but the output it gives us is similar to the hand-computed output we found at the end of our last example.

Once we've selected our candidate SSTables, we assemble the iterators, which materialize the output SSTable. As we read through the iterator, we write the new SSTable into the larger level of the levels we've chosen.

Then, we save the new state of the LSM Tree (level information, etc) in some kind of persistent state, such that the tree knows to look for this SSTable as a part of this level.

Lastly, we delete the old SSTables in the tree to reclaim our disk space.

As you can see, we've maintained the invariant that all levels of the table (except for level 0) have non-overlapping key ranges.

A good question to think about is: what do we do with tombstones during the compaction process?

Types of compaction

Now that we know that compaction also serves the process of maintaining not just data size but also the index by which we serve reads (the files, the levels, and the metadata for each level), there are multiple types of compaction strategies, each of which have different effects on lookup cost. There are multiple strategies present today, and this is an area of active research; A paper titled "Constructing and Analyzing the LSM Compaction Design Space"³ dives much deeper into this topic, and would be a good read for anyone trying to get a comprehensive view.

I won't be covering more compaction strategies in this post, but i'll probably expand upon this later.

Footnotes

It's symphony. In a database, all the components need to work together at extremely high speed and in exactly the right way; the kernel, the RAM, the network sockets, the buffers, the consensus algorithm, the scheduler, the parser, the compiler... It's like an orchestra, or a death metal band

There's a certain beauty to the unseen demons that show up at high speeds; high concurrency and high throughput is like the scream of the wind in your ears when you push your bike past 100.

And sometimes the most unlikely shit works; think of the Cannons, in Tchaikovsky's 1812 overture. No orchestra was supposed to have Cannons, but it works!

This paper deals with MapReduce, a way to perform large-scale computations with the fundamental operations (and functional primitives) map and reduce, in a distributed, parallelized manner.

If you're unfamiliar with the concepts here, I introduce the problem, and some basic primitives, in the background section - Keep reading for that! If you already know what these primitives are, and want to get to the paper directly, click here.

Table of Contents

1. Background
2. The Paper's Contributions
3. Implementation
4. Execution
5. Metrics, Benchmarks and Performance
6. MapReduce at Google
7. Assumptions and limitations
8. Conclusion

Background

To search the World Wide Web, one must first know what's on the web. To that end, one must generate a index - a data structure that holds both an item, and it's location.

In the same way the index of a book holds the name of the chapter and the page that chapter begins on, if one wishes to search the big ol' WWW by, say, a word, they'll need to maintain an index consisting of the item (the word being searched for) and its location (the URL of the page where the word is found).

Thus, creating an index of the WWW is the problem that Google faced in their early days. Way back then, the internet was (and mostly still is) a collection of HTML Documents maintaining links to each other by means of Hyperlinks (now URLs).

To generate an index, Google had to process as many parts of the web (as many documents) as they could, and perform said computations on it. However, this wasn't an easy problem at all - by around 1998, when Google launched, 512MB of RAM was a luxury, and there were already 2.4 Million websites on the internet ¹. There were a lot of documents to process.

At the time, Engineers at Google had already solved this problem, by hand-rolling their own, case-specific implementations of a computing system for each task (indexing the web, running the PageRank Algorithm, and so on). They'd get hundreds or thousands of computers together on a network, get them to talk to each other, and co-ordinate these large tasks.

This was the only way these tasks would finish in a reasonable amount of time. However, these implementations ran into various issues, such as frequently failing disks, tasks that took too long to run, and so on.

Around this time, the authors of the paper noticed that most such compute-intensive tasks they ran at Google comprised of a series of map and reduce operations.

Map and Reduce are both primitives of Functional Programming, that allow you to use functions as arguments to another function.

Map

The map function takes in an array and a function, and it applies that function to every element of the array.

>>> def square(x):
...     return x * x
>>> a = [1, 2, 3, 4, 5]
>>> list(map(square, a))
[1, 4, 9, 16, 25]

Reduce

The reduce function takes an array and a function, and uses that function to combine the elements of the array.

>>> from functools import reduce
>>> def add(a, b):
...     return a + b
...
>>> a = [1, 2, 3, 4, 5]
>>> reduce(add, a)
15

Keep in mind that these map and reduce functions can be anything!

The Paper's Contributions

I see the paper having made two major contributions -

1. Programs written as a series of Map and Reduce operations (and there are many that can be written this way) can be easily parallelized and distributed while maintaining an easy, simple interface

A simple example of a MapReduce program to find the sum of twice a list of numbers

An example of how individual computers can handle map and reduce operations, enabling high parallelism and performance

Note that this paradigm consists of users only providing the functions that will be run on the map and reduce operations. They needn't worry about how or where it's implemented - they were inherently parallelizeable and could be distributed across multiple computers.

2. A high-performance implementation of this interface on a large cluster of commodity computers

Google implemented a MapReduce system on a large cluster of commodity computers (~1000), which allowed them to complete indexing the internet in only seven passes of MapReduce.

The Programming Model

For the map function, the user provided a function that would take in some arguments (usually two, referred to as an "input pair"), and which emitted a series of intermediate key-value pairs.

The reduce function must then be written to accept a specific key, and a set of values for that key. It then reduces them to either form

1. A smaller set of values, or
2. One output value

From the paper, this can be summarized as the following - MapReduce functions described as types, taken from the paper

The map function takes in a key-value pair, and emits a list of key-value pairs. One can think of these keys emitted as "intermediate keys".

The reduce function takes in an intermediate key and a list of values for the same key, and emits a list of values (which can also have only one value).

The intermediate keys serve to help create a separation between different reduce jobs. Reduce Jobs for the Intermediate key "I" will only handle values coming which have been emitted with the Intermediate key "I".

Google's implementation uses some very clever tricks to make sure that intermediate key-value pairs for a key A emitted by the map job end up at the system running the reduce job for the same key. We'll explore this some more soon.

Some Programs in Google's MapReduce

Let's start off with the simplest example.

A MapReduce program to count the number of times each word occurs in a document or a string, taken from the paper

Here, emitIntermediate is a function used to emit an Intermediate key-value pair. emit is used to emit the final value after the reduce job completes.

Note that it's possible to perform many, many types of computation with this model. The paper mentions a few, one of which we'll take a look at here. Apart from the one here, the paper also mentions

• A Distributed Text Search
• URL Access Frequency Count
• Term-Vector per Host
• Inverted Index
• Distributed Sorting

Reverse Web-Link Graph with MapReduce

A simple example of four webpages, linking to each other

Let's say you want to find a list of pages (call them sources) that link to a specific page (the target) via a hyperlink.

This might be important, because you might want to rank how "important" a page is by seeing how many people link to it. Let's look at the page C.

Here's what our map function looks like -

def map(target, webpage):
  # check all the links in the page
  for link in webpage:
      # emitIntermediate(target, source)
      emitIntermediate(link, webpage.url)

Here are our intermediate values generated -

[(b, a), (d, a), (c, a), (c, b), (c, d), (a, c)]

Note that we're using the target link as our intermediate key.

Implementation Magic makes available to each reduce function the following -

[
(b, [a]),
(a, [c]),
(c, [a, b, d]),
(d, [a]),
]

Now, here's reduce -

def reduce(key, list):
    count = 0
    for link in list:
        count += 1
    emit(key, count)

Once reduce is called for all pages [a, b, c, d], we get -

[
(c, 3),
(a, 1),
(b, 1),
(c, 1)
]

This tells us that c is the most "important" page.

Implementation

Google's implementation, as suggested in the paper

Now that we know how to implement some common programs in MapReduce, let's see how Google did it.

It all starts with a Master (we'll call this the Supervisor) process or worker node, which controls, schedules and orchestrates multiple MapReduce jobs on a cluster of commodity machines, each known as a worker node (we'll call these the Task nodes).

The task nodes can be threads, processes, or independent computers. For now, we'll only consider the last aspect.

Every task node is considered a generic task node, and isn't limited to being used only in the map or reduce phases of the entire process.

The supervisor node controls what job to schedule on which task node and when, so a node which ran map, if free, can also be used to run reduce.

The Scheduler also takes care of splitting data by key, or by size, to make it easier to transfer.

Execution

Execution overview - MapReduce

1. The data to be processed is split into M pieces of 16Mb - 64Mb blocks. This block size can be controlled by the user. At this same time, the program is started on multiple machines.

2. The Supervisor program is spawned, and it assigns work to all the Task Nodes. The Supervisor node also assigns idle nodes to different MapReduce Jobs.

Execution Overview - Supervisor Node

3. On the map side
   1. Read the input assigned
   2. Parse the Key-Value pairs and pass them to the map function
   3. Intermediate key-value pairs are buffered in-memory
   4. These buffered pairs are written to disk at regular intervals, and they're written partitioned by the key.
   5. The locations of these pairs are passed to the Supervisor node.

Execution Overview - Map Task Node

4. On the reduce side
   1. When a reduce job is scheduled, it uses a Remote Procedure Call to fetch the buffered pairs from the map worker.
   2. Once all intermediate data has been obtained, the reduce task node SORTS the key-value pairs to ensure the same pairs are grouped together.
   3. The task node iterates over the data, passes the key-value pairs to the reduce function.
   4. The Output from this is appended to the final output file.

Execution Overview - Reduce Task Node

5. The task is complete!

A (very cool) Note on Locality

One of Google's primary concerns was the reduction of network bandwidth being used, as it was a bottleneck. To prevent this from happening, Google implemented the Google File System (the GFS) - a distributed filesystem - to ensure that data was where it needed to be before it was required.

How the GFS works internally is perhaps the subject of another blog post itself, so we won't be concerning ourselves with that right now.

GFS (Google File System) handles moving files around, with a focus on ensuring that as much data is stored locally as possible. It also relies on replication - by dividing each file into 64MB Blocks, and ensuring that (typically) at least 3 copies of each block are available on different Task Nodes.

The coolest part is that the Supervisor Node is aware of this replication and locality effort, and attempts to schedule tasks on Task Nodes which already have the input files necessary to run the task. If one isn't available, it places the task on a machine close to (i.e. in the same network as) the replica. This way, bandwidth doesn't need to be wasted to move blocks back and forth.

A smaller note on Supervisor Data Structures

The supervisor node utilizies a number of data structures to keep track of all the tasks present in the cluster.

For every map and reduce task, it stores state (IDLE | INPROGRESS | COMPLETED) and the identity of task nodes.

For every completed map task, it stores the locations and sizes of the intermediate files produced. Updates happen as and when files are added or written to, and the same information is propagated to the reduce workers (this is only for INPROGRESS tasks)

Fault Tolerance

Such massive distributed systems can fail for multiple reasons. These are known as "Failure Modes", and there are many possible cases. The sheer scale of operation implies that failure is a regular event, not a special edge case.

If A Task Node Fails...

The Supervisor pings every Task Node periodically to ensure it's still alive. This is standard practice in most systems like this.

If a Task Node is dead (i.e. has not responded to the Supervisor after a threshold amount of time), all the tasks assigned (both completed and inprogress) to it get marked as IDLE and thus become eligible to be allocated to another node in the cluster.

Even the COMPLETED tasks are marked as IDLE as a Task Node stores its intermediate K-V pairs locally - if the node is inaccessible, then so are these intermediate K-V Pairs.

If The Supervisor Node Fails...

The Supervisor data structures written to disk, if a master fails, it's easy to restore it at that current state

Atomicity and Semantics in the presence of failures

The paper makes the following statement -

"When the user-supplied map and reduce operators are deterministic functions of their input values, our distributed implementation produces the same output as would have been produced by a non-faulting sequential execution of the entire program."

This is accomplished by using periodic, atomic writes of both the map and reduce outputs on their Task Nodes.

Since the writes are atomic, and the outputs are deterministic, even multiple runs of the same task will give the same output. Since it's atomic, there won't be any partial writes.

Dealing with Slower Machines by using Backup Tasks

Some nodes are stragglers - they become slow and hold up the scheduling queue, while completing one or a few of the LAST COUPLE OF REMAINING MapReduce tasks in a cluster. This slows down completion.

When an operation is close to completion, the Supervisor system schedules backup executions of the remaining INPROGRESS tasks, so at least one version will complete. This scheduling is tuned to increase the overall workload by only a few per cent.

Metrics, Benchmarks and Performance

The benchmarks for this cluster were considered mainly for two tasks -

1. Sorting 1 TB of Data
2. Searching 1 TB of Data

They were performed on a MapReduce cluster comprising roughly 1,800 computers, with 2x 2GHz Intel Xeon Processors, 4GB RAM, and 160GB IDE Disks per machine.

They find that Backup task handling proves to be crucial, absence of which proves to increase time by an average of 44%. The cluster is relatively resistant (only 5% delay) to machine failures, or process failures, at an 11.454% failure rate.

Sorting

Data Transfer rate as a heuristic to estimate throughput of task completion. Notice how not having backup tasks makes a significant difference - Taken from the Paper

Each row in the above graph represents a different phase of the sorting task, where

• input - data read into the map jobs
• shuffle - buffered pairs from map jobs sent to reduce jobs and sorted
• output - reduce tasks complete and write to final output files

And each column corresponds to a different cluster configuration, being

• Normal Execution (with Backup Tasks)
• No Backup Tasks
• With 200 Tasks being killed off

This was modelled after the TeraSort benchmark. Since the reduce function sorts by keys, the Distributed Sort is fairly simple to implement in MapReduce.

Assuming a pre-existing knowledge of key distribution, Google's sorting implementation completed the sorting in 891 Seconds, as opposed to the current record, coming in at 1057 Seconds.

Searching

The implementation searched through 1 TB data in ~150 seconds, including around 60 seconds of startup time. Some delays and overheads were due to -

• Interactions with GFS
• getting the program to newly assigned machines

MapReduce at Google

A graph showing the usage of MapReduce at Google

This was obviously a game-changer for Google, who began to use MapReduce not only for their production indexing system, but also for multiple other Google Services.

Their production indexing system crawled through 20TB of Data using MapReduce, with around Five to Ten passes of MapReduce for the entire indexing job. The same system went from around 3800 Lines of C++, to around 700 Lines when used with MapReduce.

Lastly, it allowed programmers who aren't distributed systems experts to take advantage of the infrastructure that Google has, as it abstracts away all the fault tolerance, parallelization, locality optimization and load-balancing details from the programmer. Since it focussed specifically on reducing network bandwidth usage, it allowed Google to deploy it at the scale we saw it run.

Assumptions and Limitations

• Assume functions being passed as Map and Reduce functions do not depend on Global State
• Assumes the Supervisor node can make scheduling decisions at scale - must make O(M + R) scheduling decisions, and must keep O(M * R) state in memory.
• Google's implementation assumes the task is restarted if the Supervisor Node fails
• No guarantee of similar performance in sorting benchmarks without prior knowledge of key distribution

Conclusion

This was my first ever paper review! If you liked it, do reach out to me to let me know what you think about it. I've got papers in mind that I want to cover, but if you find a paper you think is really cool, please don't hesitate to reach out!

As a paper, this one was super well-written, and I found my questions having been answered before I knew I had them. 10/10 recommended read.

Why?

Git is an essential, industry-standard Version Control System (VCS) - In essence, Git is how you work on the same thing (usually a document or a bunch of files) with multiple people at the same time, without needing to break your head about it.

Remember when you'd save your work at different points in time by naming things roughly this way?

Fear not, my friend! Gone are the days where you'd have to do this. Pick your jaws off the floor, and let me introduce you to Git, which saves you from having to resort to this... practice :D

What is Git?

Git is a Distributed (i.e. can run on multiple connected computers) Version Control System (VCS) that helps you

• track changes made to your files
• work on multiple different versions of your files at once (no, I'm not kidding! You can!)
• revert to an old version of your files
• collaborate with other people on the same files

For all practical means and purposes, Git itself is a piece of software that was written to help the maintainers of the large software projects (Git was written by Linus Torvalds and Junio Hamano, both prominent maintainers of the Linux Kernel) do their work easily.

Isn't it amazing that you can use this tool regardless of the scale of your project?

Let's get learning!

Let's Talk about Maggi.

I like Maggi Noodles.. I think they're excellent, not just as noodles, but as a "meal platform" of sorts - it's a base on which you can build a lot more dishes, much like bread.

The beauty of Maggi (and most noodles) is that we can either have them plain, or we can customize them by adding our own ingredients.

In the quest for the ultimate Maggi Noodle recipe, you begin to keep track of all the recipes you try out.

To do this, you take the help of a Logbook, and a rough piece of paper.

Some Basic Rules

Before we embark on the wild quest to find the best maggi recipe known to humankind, we set the following rules in place -

1. Consider a Recipe to be a Collection of Ingredients and Quantities.

   This helps us quantify things! A recipe can look like this -

   NOODLES   10
   MASALA    20
   TOMATOES  03
   

2. Each and Every change you make is either an ADDITION (+), DELETION (-) or a MODIFICATION (~) of said ingredients and their quantities.

   Let's say I started off with the recipe above. I liked it, but I felt like it could use more spice and one less tomato, so the difference would look like this -

   + SPICE   10
   - TOMATO  01
   

   Applying this transformation on the recipe, the new recipe looks something like this -

   NOODLES   10
   MASALA    20
   TOMATOES  02
   SPICE     10
   

3. The Logbook keeps tracks of versions of recipes by keeping track of the changes.

   At this point, it's easy to see how to undo a change, simply by inverting the signs! Let's say we want to get back to the previous version. The Difference for this change looks like this -

   - SPICE   10
   + TOMATO  01
   

   Magic! We're back where we started.

   The logbook only keeps track of all your changes.

4. When you're done fiddling around with a recipe on the rough sheet, you write down the changes in the logbook, recording it as a version.

   Once we're done experimenting, and we like how a certain change to the recipe tastes, we write it down in the logbook!

   Our logbook looks something like this -

   # Initial Version
   + NOODLES   10
   + MASALA    20
   + TOMATOES  03
   
   # version 1
   + SPICE   10
   - TOMATO  01
   

Whew! That was a bit. Take a minute to make sure your understand what's going on! You'll thank yourself later, I promise you.

It's time to get cooking!

What's Going On!?

So, you're making a batch of the latest experimental Maggi. You'd obviously want to know how you're changing your recipe, so you write down all your changes on the rough paper before you decide you like it, and so that you can easily modify it.

At any given point of time during our experimentation process, we want to see what's going on in our recipe book. Here's what this set of actions looks like -

Eventual Comfort

So you've decided on a set of tasty tweaks you want! Here's the rough procedure you follow to immortalize the latest reels-compliant and SEO-Friendly version of your Maggi -

The Situation So Far

After a fun day of experimentation, here's what your logbook may look like.

I could say more about Tobasco sauce (but I digress).

A Friend wants in!

Having heard about all the fun you've been having with your Maggi, a friend - Ramesh - Wants in!

You tell Ramesh -

Hey, Here's the system I'm following! Here's why it works, and here's how you can use it!

So, in the true spirit of Open Source, you decide to give Ramesh your recipe collection and logbook to start his own, so that he can see your history of experimentation, and add his own changes, too!

Since you trust Ramesh to make some nice additions, you decide to frequently incorporate his changes to the recipe into yours.

Together, your recipes shall RULE THE WORLD! (and also only incidentally taste very nice)

Now, since your logbook is just paper, you can simply copy it all at once, and have multiple copies exist at the same time!

Let's see how we'd go about making a copy and working on it in parallel -

Solid. Both Ramesh and you can work on the logbook at the same time (in essence) and merge your changes later to reach a new, final version.

A Bird's eye view of this process looks something like this -

Keep in mind that you can have any copy of the logbook be the starting point of an entirely different copy, so it's possible for us to have an arbitrary configuration of these "branches". Think of all the cool diagrams!

How does this combination happen?

Since both versions originate from a single point, there's always the possibility you can combine the changes from both versions! (remember this.)

Your change can look like this -

+ SPICE   10
- TOMATO  1

And Ramesh's, like this -

+ SALT     11

Combined, these changes will work the same way!

+ SPICE    10
- TOMATO   1
+ SALT     11

What if you both make try to change the same thing?

What if you said

+ SALT     10

And Ramesh Said -

- SALT     5

Our Logbook doesn't know how to combine these two changes! You'll have to either give it an entirely new value, or you'll need to decide whose version to keep, and whose version to discard.

Communication is important.

This leaves us in a funny place - unless you and Ramesh agree upon what to change and what not to change beforehand, this situation is common!

It's formally known as a merge conflict, and can be pretty scary, but fear not! You'll get used to resolving them in no time.

Congratulations! You know know how Git Works.

🥳️

Wait, what?

Let's make things simpler - it turns out we were talking about code and version control all along!

Let's map our example to Git.

In reality, you can apply more or less the same recipe-and-logbook model to software, code and anything you can represent in files!

So far, using this model, we learnt how to -

• track changes made to your recipes files
• work on multiple different versions of your recipes files at once
• revert to an old version of your recipes files
• collaborate with other people on the same recipes files

Sure, okay, I have something of an understanding - but how do I use it? Isn't Git a Command Line tool?

Fear Not! Let's see how these procedures we learnt for our logbook map to Git.

So, Get out your terminals (Git Bash, or just your regular terminal), and make sure you've got git installed. You can download and install it from here - the default options are almost always good enough.

What's Going On!?

Here's how you can understand what's going on in your repository, in the same context of the cookbook -

Let's start off by creating an empty repository. We'll then add just one file to it for now, and we'll make changes in it.

NOTE: Anything after the $ is a command.

$ git init
Initialized empty Git repository in /home/anirudh/git_101/.git/

$ ls -la
total 20
drwxrwxr-x   3 anirudh anirudh  4096 Feb  2 00:08 .
drwxr-x--- 131 anirudh anirudh 12288 Feb  1 23:26 ..
drwxrwxr-x   7 anirudh anirudh  4096 Feb  2 00:08 .git

We see that the folder is entirely blank, save for the .git folder. This is where the entire repository lives, and it holds everything git needs to know about your repository. Let's see what's going on -

$ git status
On branch main

No commits yet

nothing to commit (create/copy files and use "git add" to track)

git status (a very useful command to use as often as you want) is telling us what we know - this is a blank folder. We're on the default branch main, and we have no commits yet. So let's do what it says, and add a file.

$ echo "Hello, world!" > hello.txt
$ cat hello.txt
Hello, world!

The first command just creates a file called hello.txt and writes "Hello, world!" into it. We then use the program cat (might not be available on windows, so use a text editor of your choice!) to verify as much. Let's run git status to see if that had any effect -

$ git status
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        hello.txt

nothing added to commit but untracked files present (use "git add" to track)

Now Git knows that there's a file (or a potential recipe on the rough sheet), and it hasn't been written to the logbook yet! We'll fix this right now.

Eventual Comfort

Once you've messed around with your code enough, and you think it's time to save it as a version, you can use these commands!

Let's start off by running git log wherever we left our repository.

$ git log
fatal: your current branch 'main' does not have any commits yet

Spot on! We'll fix this by following the steps above.

$ git status
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        hello.txt

nothing added to commit but untracked files present (use "git add" to track)

The output here's remained the same. Let's now use the git add command to add all our files to the staging area.

$ git add -A
$ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   hello.txt

We use the -A flag here to specify that we want to track all files, so as to not have to specify files individually (you can do that if you wish to!).

Running git status immediately after shows us that Git knows, and is keeping track of, this file. It's now ready to write it into our logbook at any point of time.

Let's do that!

$ git commit -m "first commit!"
[main (root-commit) 39e97d5] first commit!
 1 file changed, 1 insertion(+)
 create mode 100644 hello.txt

A little involved here - the -m "<message here>" is followed to create a "commit message" for each commit. Think of it as leaving a note explaining your changes to anyone else who sees this.

DO NOT WORRY if the numbers you see aren't the same - they're what are known as commit hashes and aren't supposed to be the same by design. Commit hashes are unique to a commit, and can be used to identify individual commits.

So you can now see that Git is, indeed, keeping track of insertions and deletions.

Lastly, let's run git log to see what's going on -

$ git log
commit 39e97d52c2c0c26a1c3a314c040c0f01f476a3bc (HEAD -> main)
Author: Anirudh Rowjee <ani.rowjee@gmail.com>
Date:   Wed Feb 2 00:11:40 2022 +0530

    first commit!

Here we can see the full commit hash, as well as the details of the person who made the commit - here's where the commit message comes into play! We can also see that the HEAD commit (the latest commit) is on the main branch.

Writing a detailed commit message is always helpful, and is a good practice.

You can have as many commits as you want!

A Friend wants in!

Here's how we'll deal with collaboration and multiple versions existing at once -

Let's start off by listing all the branches our repository has.

$ git branch --list
* main

The * tells us which branch we're currently on. Good IDEs, good text editors and good shells will go out of their way to make the current git branch obvious to you.

At the moment, only one, but who said we're done? Let's make a new branch.

$ git branch new_branch
$ git branch --list
* main
  new_branch

Let's switch to the branch we just created.

$ git checkout new_branch
Switched to branch 'new_branch'
$ git branch --list
  main
* new_branch

Psst! You can both create and switch to a new branch at the same time by using the -b flag with git checkout - the above could've been accomplished with just git checkout -b new_branch

Now we make some changes to this file, and add a commit or two.

$ nano hello.txt

$ cat hello.txt
Hello, world!


This is Anirudh, from the new branch!

Once we've made changes to the file, we're ready to commit.

$ git add -A

$ git status
On branch new_branch
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   hello.txt

$ git commit -m "made change on branch"
[new_branch 5128622] made change on branch
 1 file changed, 3 insertions(+)

Let's switch to our main branch, and take a look at the file.

$ git checkout main
Switched to branch 'main'

$ cat hello.txt
Hello, world!

No change! We now have two versions of our file existing at the same time on two branches.

$ git checkout new_branch
Switched to branch 'new_branch'

$ cat hello.txt
Hello, world!

This is me from a new branch. HI!

Quickly checking git log tells us the story of two branches -

$ git log
commit 51286222c4e75db4ffe04f9284d4af19f348a56f (HEAD -> new_branch)
Author: Anirudh Rowjee <ani.rowjee@gmail.com>
Date:   Wed Feb 2 00:16:12 2022 +0530

    made change on branch

commit 39e97d52c2c0c26a1c3a314c040c0f01f476a3bc (main)
Author: Anirudh Rowjee <ani.rowjee@gmail.com>
Date:   Wed Feb 2 00:11:40 2022 +0530

    first commit!

HEAD points to the latest commit in the repository. For the new_branch branch, it's the one where we made a change!

$ git checkout main
Switched to branch 'main'

$ git log
commit 39e97d52c2c0c26a1c3a314c040c0f01f476a3bc (HEAD -> main)
Author: Anirudh Rowjee <ani.rowjee@gmail.com>
Date:   Wed Feb 2 00:11:40 2022 +0530

    first commit!

Since main doesn't know about the changes made in new_branch, HEAD points to a different, earlier commit.

How all this works under the hood is the combination of years of experience and some frankly amazing engineering, which is most definitely out of the scope of this article. Maybe Another Day!

Now, Let's merge our two branches to see what happens to the file!

$ git merge new_branch
Updating 39e97d5..5128622
Fast-forward
 hello.txt | 3 +++
 1 file changed, 3 insertions(+)

WOOHOO! Just a quick check confirms that the changes from our branch, have, indeed, been updated into the main branch -

$ cat hello.txt
Hello, world!


This is Anirudh, from the new branch!

Taking a look at git log confirms as much.

$ git log
commit 51286222c4e75db4ffe04f9284d4af19f348a56f (HEAD -> main, new_branch)
Author: Anirudh Rowjee <ani.rowjee@gmail.com>
Date:   Wed Feb 2 00:16:12 2022 +0530

    made change on branch

commit 39e97d52c2c0c26a1c3a314c040c0f01f476a3bc
Author: Anirudh Rowjee <ani.rowjee@gmail.com>
Date:   Wed Feb 2 00:11:40 2022 +0530

    first commit!

As we can see, HEAD points to both our branches.

If we're on the branch we want to merge into, we simply need to use this command to merge changes from the branch to the main branch.

$ git merge <branch_name>

This about sums it up! You're now ready to use Git, and you know enough to debug any issues you might run into with this, or at least use google to debug better!

The Git Cheat Sheet by Anirudh Rowjee

Feel free to take a screenshot of this, and keep it around as a handy reference :D

What now?

You've made it this far, so pat yourself on the back!

If you want to learn more, the official documentation is an excellent place to start.

I hope you now have some idea of how to use Git, given that you have some understanding of what's going on apart from the output you see from the commands. I'd urge you to use Git for your personal projects (if you don't already), so you can get used to using this awesome tool!

Thank you for your time, and I hope I've been of some help!

Oh, and don't shy away from telling your friends about this if it helped you :D

PS: I'd love to hear from you if you found this helpful - reach out through my twitter or email (found at the bottom of the page). Thank you!