I built a storage engine in Go (from scratch without any AI), here's the entire process documented.

I spent the last 2 months building a storage engine from scratch understand how storage engines actually work. To respect the rules of this subreddit, we will not discuss the features ør benchmarks, for that you should read the repo yourself. To understand how I came up with the bits and pieces to put together this project, follow along. So here's my entire journey. Two things: - I built this project because of a recent interest in low-level and systems programming, that combined with my general affinity towards stateful systems made this project an obvious choice. - I see a lot of cool projects (on various social platforms) and when I attempt to read their code, it's obvious that it's written by AI. I intend to share my thought process here because I want to spread awareness that it's very much possible to build something like a storage engine using your first principles intuition. ---------------- Let's begin: 1 . Given all the knowledge I had and using first principles thinking, I setup an in-memory KV store but then it's obvious I had to make it persistent. I added persistence by writing a single line to the file every time someone made a PUT request. The file now had a bunch of {key: "hello", value: "world"}\n. So during startup, I would read all these lines and recover everything into in-memory. 2 . For me, at this point that's all the upfront knowledge I had. So I asked some very basic questions: > How would I recover the entire file into memory on startup? At some point it just wouldn't be possible because the file is growing unbounded. This means that I must not load everything in-memory and instead access the file directly > But then if I read every line top to bottom on every GET then my latency would be literally obliterated? This means I must somehow efficiently query the file. I came up with a solution, I created files based on alphabets, all keys with prefix A will end up in file A, all keys with prefix B will end up in file B and so on. By first prin