Database Learning Path

A self-paced, hands-on curriculum for mastering database internals — with a focus on performance, data structures, and algorithms — built around reading world-class codebases, implementing things from scratch in Rust, and benchmarking everything.

Background: author is a core developer of FalkorDB and falkordb-rs-next-gen, so graph internals are familiar ground; the goal is breadth + depth across all database domains.

Read this online

The whole path is published as a browsable book (sidebar navigation, search, rendered diagrams) at https://aviavni.github.io/database-learning-path/ — or download the entire book as a PDF.

To build locally: cargo install mdbook mdbook-mermaid, then mdbook-mermaid install . && mdbook serve from the repo root.

Repo layout

README.md          ← you are here: how this repo works
PLAN.md            ← the full curriculum (32 topics) — the source of truth
PROGRESS.md        ← status tracker: what's done, in progress, next
capstone/          ← falkordb-rs-next-gen rebuilt from scratch, one milestone per topic
topics/            ← created lazily — one dir per topic when the deep dive starts
  NN-topic-name/
    README.md      ← expanded study guide for that topic
    notes.md       ← learnings, insights, surprising things
    experiments/   ← standalone Rust experiments + criterion benchmarks
resources/
  papers.md        ← papers & articles (arXiv, VLDB, SIGMOD, classics)
  codebases.md     ← reference codebases and what each is best for studying
  tools.md         ← profiling, benchmarking, fuzzing, testing tools

Workflow (for me and for Claude next session)

flowchart LR
    A["PROGRESS.md<br/>where are we?"] --> B["PLAN.md<br/>pick next topic"]
    B --> C["topics/NN-name/<br/>expand into study guide"]
    C --> D["study · experiments<br/>· benchmarks · notes.md"]
    D --> E["capstone/<br/>milestone MNN"]
    E --> F["update PROGRESS.md<br/>+ commit"]
    F -.->|next topic| B

1. Open PROGRESS.md to see where we are.
2. Pick the next topic (or any topic — order is a suggestion, not a rule) from PLAN.md.
3. Create topics/NN-topic-name/ and expand the PLAN.md section into a full study guide: concept explanations with examples, guided code-reading of the reference repos, exercises, and benchmarks.
4. Study, implement experiments, benchmark, take notes in notes.md.
5. Implement the topic’s capstone milestone in capstone/.
6. Update PROGRESS.md (status + one-line takeaway) and commit.

Conventions

• Language: Rust for all implementations and benchmarks (criterion + flamegraph).
• Every topic ends with a benchmark — numbers over intuition.
• Code reading is done against pinned clones under ~/repos/ (not vendored here).
• Notes capture why designs win, trade-offs, and measured results — not summaries.

The Plan — Database Internals Curriculum

32 topics, self-paced, deliberately diverse: storage / in-memory / query / graph / vector / distributed / hardware topics are interleaved so it stays fun. Each topic has: why it matters, core concepts, reference code to read, key papers, and a build+bench exercise that also advances the capstone (capstone/README.md).

Order is a recommendation. Topics 0–6 are the foundation; after that, jump around freely.

The map

flowchart TD
    subgraph FOUNDATION["Foundation — do in order"]
        direction LR
        T0["0 perf<br/>toolbox"] --> T1["1 B-tree<br/>vs LSM"] --> T2["2 in-memory<br/>structures"] --> T3["3 B-tree<br/>internals"] --> T4["4 LSM<br/>deep dive"] --> T5["5 WAL &<br/>recovery"] --> T6["6 buffer<br/>pool"]
    end
    subgraph CORE["Systems core"]
        T7["7 networking"]
        T8["8 MVCC"]
        T9["9 concurrency"]
    end
    subgraph QUERY["Query engine"]
        T10["10 parse/plan/optimize"] --> T11["11 execution models"] --> T19["19 JIT"]
        T12["12 columnar"]
    end
    subgraph GRAPH["Graph — home turf"]
        T13["13 graph engines"] --> T20["20 GraphBLAS internals"] --> T24["24 graph algorithms"] --> T25["25 graph ML"]
    end
    subgraph SEARCH["Indexes & search"]
        T14["14 vector"]
        T23["23 full-text"]
        T26["26 probabilistic"]
    end
    subgraph HW["Hardware"]
        T17["17 SIMD"] --> T18["18 GPU"]
    end
    subgraph DIST["Distributed"]
        T15["15 replication/Raft"] --> T29["29 distributed txns"]
        T28["28 cloud-native"]
        T31["31 CRDTs"]
    end
    subgraph CORRECT["Correctness"]
        T16["16 testing"] --> T21["21 formal methods"]
    end
    subgraph STREAM["Streaming & temporal"]
        T27["27 incremental views"]
        T30["30 time-series"]
        T27 --> T32["32 HTAP"]
    end
    T22["22 standard benchmarks — the yardstick for everything"]
    FOUNDATION --> CORE
    FOUNDATION --> QUERY
    FOUNDATION --> GRAPH
    FOUNDATION --> SEARCH
    FOUNDATION --> HW
    FOUNDATION --> DIST
    FOUNDATION --> CORRECT
    FOUNDATION --> STREAM
    QUERY --> T22
    GRAPH --> T22

0. The Performance Toolbox

Why: You care about performance — so learn to measure before learning to build. Everything after this topic gets benchmarked properly.

• Concepts: microbenchmark pitfalls (warmup, variance, coordinated omission), CPU caches & memory hierarchy, branch prediction, TLB, perf counters, flamegraphs, latency percentiles vs throughput, roofline thinking.
• Read code: criterion.rs internals (how it fights noise), RocksDB db_bench, redis redis-benchmark.c.
• Papers/reading: “Systems Performance” (Gregg) ch. 1–2; “Fair Benchmarking Considered Difficult” (DBTest ’18); “How NOT to Measure Latency” (Tene talk); Drepper §3–4.
• Build & bench: Rust bench harness comparing Vec scan vs HashMap lookup vs BTreeMap across sizes; produce flamegraphs; observe cache-line effects (seq vs random access).
• Capstone milestone M0: scaffold the falkordb-scratch workspace + criterion bench harness + graph workload generator; record baseline numbers from the real falkordb-rs-next-gen to chase.

1. Storage Engine Landscape: B-Tree vs LSM

Why: The single most consequential design decision in a database. Frames everything else.

• Concepts: read/write/space amplification triangle (RUM conjecture), in-place vs out-of-place updates, page-oriented vs log-structured, where each engine family wins.
• Read code: fjall (small, clean Rust LSM), turso (core/storage/ — SQLite-style B-tree in Rust), tidesdb (C LSM), RocksDB high-level layout.
• Papers: “The LSM-Tree” (O’Neil ’96), “The Ubiquitous B-Tree” (Comer ’79), “Designing Access Methods: The RUM Conjecture” (2016), “Architecture of a Database System” (Hellerstein/Stonebraker).
• Build & bench: benchmark fjall vs a raw B-tree (e.g. redb/sled) on write-heavy vs read-heavy vs scan workloads; explain results in terms of amplification.
• Capstone M1: define the storage-backend abstraction (compare with the reference’s graph/src/storage/backend.rs after designing yours) — in-memory first, persistent backends swap in later.

2. In-Memory Structures: Hash Tables, Skip Lists, Tries

Why: Redis’s dict and FalkorDB’s core structures — the workhorses of every in-memory DB.

• Concepts: open addressing vs chaining, incremental rehashing (redis), SwissTable/SIMD probing (hashbrown), skip lists (why LSM memtables use them), radix trees / ART, cache-conscious layout.
• Read code: redis dict.c (incremental rehash!) + valkey’s changes, redis t_zset.c (skiplist), hashbrown, RocksDB memtable/ (concurrent skiplist), redis rax.c (radix tree).
• Papers: “The Adaptive Radix Tree” (Leis ICDE’13), Google SwissTable talk (CppCon 2017).
• Build & bench: implement a skip list and an incremental-rehash hash table in Rust; bench vs hashbrown and crossbeam-skiplist; measure rehash latency spikes vs redis-style incremental approach.
• Capstone M2: attribute store + string pool + node/edge ID datablocks (hash index + interning) — the reference’s attribute_store.rs/string_pool.rs, your way.

3. B-Tree Internals & Paged Storage

Why: SQLite/Postgres/LMDB/most-embedded-DBs. Pages are how disks think.

• Concepts: slotted pages, node splits/merges, B+tree vs B-tree, prefix compression, copy-on-write B-trees (LMDB), overflow pages, page checksums, varint encoding.
• Read code: turso core/storage/btree.rs + pager (Rust re-implementation of SQLite — ideal), SQLite btree.c (the classic), LMDB mdb.c (COW).
• Papers: “Modern B-Tree Techniques” (Graefe — the survey), SQLite file-format doc.
• Build & bench: implement a slotted-page disk B+tree in Rust (fixed 4KB pages); bench point lookups & range scans vs redb; try prefix truncation and measure.
• Capstone M3: disk-backed B+tree backend for properties + range indexes behind the storage abstraction.

4. LSM-Tree Deep Dive

Why: RocksDB powers half the industry (including graph DBs like TiKV-based ones). Compaction is a fascinating scheduling problem.

• Concepts: memtable→SST lifecycle, leveled vs tiered vs FIFO compaction, bloom filters (and Monkey’s optimal allocation), fractional cascading, compaction debt/write stalls, SST formats & block cache.
• Read code: fjall (read it ALL — it’s small), RocksDB db/compaction/, table/block_based/.
• Papers: “Monkey: Optimal Navigable Key-Value Store” (SIGMOD’17), “Dostoevsky” (SIGMOD’18), RocksDB paper (TODS’21), “Constructing and Analyzing the LSM Compaction Design Space” (VLDB’21).
• Build & bench: implement a mini-LSM (memtable + SSTs + leveled compaction + bloom filters) — optionally follow skyzh/mini-lsm course; measure write amp with different compaction strategies.
• Capstone M4: LSM-backed alternative persistence (graph snapshots as SSTs); benchmark B+tree vs LSM backends on graph mutation + bulk-load workloads.

5. Durability: WAL, fsync, Crash Recovery

Why: The hardest part to get right. Where correctness meets performance.

• Concepts: write-ahead logging, ARIES (redo/undo, LSNs, fuzzy checkpoints), group commit, fsync vs fdatasync vs O_DIRECT, torn pages (full-page writes / double-write buffer), io_uring.
• Read code: postgres xlog.c (skim, it’s huge), turso WAL, redis AOF (aof.c) vs RDB, RocksDB WAL.
• Papers: “ARIES” (Mohan ’92 — read a summary first, then the paper), “Scalability of write-ahead logging on multicore” (Aether, VLDB’10).
• Build & bench: add WAL + crash recovery to your B+tree; write a crash-injection test (kill -9 mid-write, verify recovery); bench fsync-per-commit vs group commit vs O_DIRECT.
• Capstone M5: WAL + crash recovery for graph mutations (contrast with FalkorDB’s reliance on redis RDB/AOF); crash-injection test suite.

6. Buffer Pool & Memory Management

Why: mmap-vs-buffer-pool is one of the great debates; allocation strategy dominates in-memory DB performance.

• Concepts: buffer pool design, eviction (LRU, CLOCK, LRU-K, 2Q), pointer swizzling (LeanStore), why mmap is (usually) wrong for DBs, jemalloc/arena allocation, NUMA.
• Read code: postgres bufmgr.c + CLOCK sweep, redis zmalloc.c, DuckDB buffer manager, LeanStore (C++).
• Papers: “Are You Sure You Want to Use MMAP in Your DBMS?” (CIDR’22), “LeanStore” (ICDE’18), “Virtual-Memory Assisted Buffer Management” (vmcache, SIGMOD’23).
• Build & bench: build a buffer pool (CLOCK) for the B+tree; bench vs mmap on datasets larger than RAM; reproduce mmap’s write-back unpredictability.
• Capstone M6: buffer pool under the persistent backends — graphs larger than RAM.

7. Networking, Protocols & Event Loops

Why: Redis’s speed is as much about the event loop and RESP as about data structures. You know the module side of FalkorDB; own the server side.

• Concepts: RESP2/RESP3 design (why so parseable), event loops (ae.c) vs thread-per-core vs async, pipelining, io-threads in redis/valkey, pgwire protocol, neo4j’s Bolt protocol (versioned handshake, PackStream binary serialization, explicit result streaming via PULL/DISCARD + cursors) — RESP vs pgwire vs Bolt as three answers to framing/typing/streaming, backpressure.
• Read code: redis ae.c + networking.c, valkey’s io-threads rework (great perf PRs to study), pgwire (Rust crate), qdrant’s gRPC/tonic setup, FalkorDB’s own src/bolt/ (it already speaks Bolt — reread it with server-side eyes).
• Papers/reading: “The C10K problem”, valkey blog posts on multithreading perf, Glauber Costa on thread-per-core, Bolt Protocol + PackStream specifications (neo4j docs).
• Build & bench: implement a RESP server in Rust (tokio) speaking GET/SET; bench with redis-benchmark and memtier_benchmark against real redis; find your bottleneck with flamegraphs.
• Capstone M7: RESP server exposing GRAPH.QUERY/GRAPH.RO_QUERY — wire-compatible with existing FalkorDB clients; bench with falkordb-py against the real thing. Stretch: a Bolt listener on a second port so neo4j drivers connect too (PackStream encoding of the graph result types).

8. Transactions & MVCC

Why: The intellectual core of OLTP. Postgres MVCC vs in-memory designs is a masterclass in trade-offs.

• Concepts: ACID, isolation levels & anomalies (read this twice), 2PL vs OCC vs MVCC, snapshot isolation & write skew, SSI, postgres tuple versioning + vacuum, HOT updates, timestamp ordering, Hekaton-style MVCC.
• Read code: postgres heapam.c + visibility rules (HeapTupleSatisfiesMVCC), surrealdb transaction layer, RocksDB utilities/transactions/.
• Papers: “A Critique of ANSI SQL Isolation Levels” (Berenson ’95), “Serializable Snapshot Isolation in PostgreSQL” (VLDB’12), “An Empirical Evaluation of In-Memory MVCC” (Wu/Pavlo VLDB’17), “Hekaton” (SIGMOD’13).
• Build & bench: implement MVCC with snapshot isolation over your KV engine; write tests that demonstrate (and then prevent) write skew; bench txn throughput vs a single global lock.
• Capstone M8: MVCC graph — copy-on-write + versioned reads (design yours, then study the reference’s mvcc_graph.rs/cow.rs).

9. Concurrency: Latches, Lock-Free & Epochs

Why: Scaling a storage engine across cores is where the hardest bugs and biggest wins live.

• Concepts: latches vs locks, lock coupling / optimistic lock coupling, lock-free structures & memory reclamation (epochs, hazard pointers), Bw-Tree, atomics & memory ordering in Rust, contention profiling.
• Read code: crossbeam-epoch, RocksDB concurrent memtable inserts, memgraph skip-list, postgres lwlock.c.
• Papers: “The Bw-Tree” (ICDE’13) + “Building a Bw-Tree Takes More Than Just Buzz Words” (SIGMOD’18 — the reality check), “Optimistic Lock Coupling” (Leis).
• Build & bench: make your skip list concurrent (epoch reclamation); bench scaling 1→16 threads; compare mutex-sharded vs lock-free; measure with perf c2c for false sharing.
• Capstone M9: threadpool + concurrent readers with single writer; parallel query execution (compare with the reference’s threadpool.rs design).

10. Query Engines I: Parsing, Planning, Optimization

Why: The optimizer is the database’s brain. Directly relevant to Cypher planning in FalkorDB.

• Concepts: logical vs physical plans, relational algebra rewrites (predicate pushdown, join reordering), cost models & cardinality estimation (where it all goes wrong), dynamic programming join ordering, Cascades framework.
• Read code: DuckDB src/optimizer/ (readable!), postgres optimizer/ (join search), sqlparser-rs, datafusion optimizer, polars lazy-frame optimizer (crates/polars-plan/).
• Papers: “Access Path Selection” (Selinger ’79 — the founding paper), “How Good Are Query Optimizers, Really?” (VLDB’15 — humbling), “The Cascades Framework” (Graefe ’95).
• Build & bench: write a mini planner: parse SQL subset → logical plan → apply pushdown + join reordering; verify plans change with table sizes; compare against DuckDB’s EXPLAIN.
• Capstone M10: Cypher-subset parser + binder + logical plan tree + rewrite rules (the reference’s parser/ + planner/ — including its optimizer dir — are your after-the-fact mirror).

11. Query Engines II: Execution Models

Why: Volcano vs vectorized vs compiled — the defining performance battle of modern analytics.

• Concepts: iterator (Volcano) model, vectorized execution (X100/DuckDB), query compilation (HyPer), morsel-driven parallelism, hash joins & aggregation internals, SIMD in query processing.
• Read code: DuckDB src/execution/ (vectors, pipelines), polars streaming engine + SIMD compute kernels (crates/polars-compute/), datafusion (Arrow-based), postgres executor/ (classic Volcano).
• Papers: “MonetDB/X100: Hyper-Pipelining Query Execution” (CIDR’05), “Everything You Always Wanted to Know About Compiled and Vectorized Queries” (VLDB’18), “Morsel-Driven Parallelism” (SIGMOD’14).
• Build & bench: implement the same aggregation query (scan+filter+group-by) three ways: tuple-at-a-time, vectorized (1024-row batches), and with SIMD; bench — the gap is the whole lesson.
• Capstone M11: vectorized runtime: batched rows + operator pipeline + expression eval (mirror of runtime/batch.rs, vectorized.rs, eval.rs).

12. Columnar Storage & Analytics

Why: DuckDB/ClickHouse-style OLAP. Compression IS performance here.

• Concepts: row vs column layout, encodings (RLE, dictionary, bit-packing, delta, FSST for strings), zone maps / min-max pruning, Parquet & Arrow formats, late materialization, columnar-store architectures compared: ClickHouse MergeTree (LSM-flavored parts + sparse primary index, materialized views) vs DuckDB (embedded, single-file) vs real-time OLAP (Pinot/Druid ingest-time indexing).
• Read code: DuckDB src/storage/compression/, ClickHouse MergeTree/ (parts, granules, sparse index — pick narrow slices), polars (Arrow memory layout in practice), arrow-rs, parquet-rs.
• Papers: “C-Store” (VLDB’05), “Integrating Compression and Execution in Column-Oriented Database Systems” (SIGMOD’06), “BtrBlocks” (SIGMOD’23), “FSST” (VLDB’20), “ClickHouse: Lightning Fast Analytics for Everyone” (VLDB’24).
• Build & bench: implement RLE + dictionary + bit-packing encoders; bench scan speed on encoded vs raw data (decompression can be faster than reading raw — verify); run ClickBench queries on DuckDB and profile.
• Capstone M12: columnar attribute storage + zone-map pruning for property filters.

13. Graph Engines (Home Turf, Deeper)

Why: Compare FalkorDB’s sparse-matrix approach against the alternatives you compete with — with benchmarks.

• Concepts: adjacency representations (CSR/CSC, adjacency lists, sparse matrices/GraphBLAS), neo4j’s fixed-size record store + pointer chasing, memgraph’s in-memory skip-list store, BFS as SpMV, worst-case optimal joins for pattern matching, LDBC benchmarks; the query-language landscape: Cypher/openCypher vs GQL (ISO/IEC 39075:2024 — the first new ISO database language since SQL) vs SQL/PGQ (property graphs inside SQL) vs SPARQL over RDF vs Gremlin vs Datalog — data models (property graph vs triples: where do edge properties go in RDF? reification/RDF-star), pattern-matching semantics (homomorphism vs isomorphism vs trail — same query, different answers!), path objects & quantified path patterns, composability (can a query’s output feed another query — Cypher’s weakness, Datalog’s strength), and what each language lets the planner push down.
• Read code: SuiteSparse:GraphBLAS internals (you know the API — go deeper into masks/complement handling), neo4j record format (kernel/impl/store/), memgraph storage/v2/, kuzu (WCOJ + columnar graph — very relevant), FalkorDB’s Cypher grammar vs the openCypher grammar spec (what’s missing/extra).
• Papers: “GraphBLAS: SuiteSparse” (Davis, TOMS), “Kùzu: A Database Management System For ‘Beyond Relational’ Workloads” (CIDR’23), “EmptyHeaded” (worst-case optimal joins on graphs), LDBC SNB spec, “Graph Pattern Matching in GQL and SQL/PGQ” (SIGMOD’22), “G-CORE: A Core for Future Graph Query Languages” (SIGMOD’18), the GQL standard overview (gqlstandards.org / Deutsch et al.).
• Build & bench: implement 2-hop neighborhood query over CSR vs adjacency-list vs GrB sparse matrix; bench on LDBC-scale data; compare with FalkorDB and neo4j on the same query; write the same three queries (filtered 2-hop, shortest path, group-by aggregation) in Cypher, GQL, SPARQL, and Gremlin — note where the language forces a different plan (path semantics, lack of pushdown) rather than just different syntax.
• Capstone M13: first graph core: adjacency-list/CSR node+edge store with basic pattern matching — the deliberately-naive baseline that M20’s sparse-matrix core will replace (and be measured against). Language-wise: target openCypher now, but keep the AST GQL-shaped (quantified path patterns as first-class) so M10’s parser doesn’t need a rewrite when GQL compatibility matters.

14. Vector Search

Why: qdrant/helix-db territory; every DB is adding this. Beautiful algorithms, very benchmarkable.

• Concepts: ANN problem & recall/latency trade-off, HNSW (and its memory hunger), IVF, product quantization, scalar/binary quantization, DiskANN/Vamana for on-disk, filtered search (the hard part — qdrant’s specialty).
• Read code: qdrant lib/segment/ (HNSW + filtering + quantization), helix-db vector side, usearch (compact HNSW).
• Papers: “HNSW” (arXiv:1603.09320), “Product Quantization” (Jégou PAMI’11), “DiskANN” (NeurIPS’19), qdrant blog on filtered HNSW.
• Build & bench: implement HNSW in Rust from the paper; measure recall@10 vs QPS curves against qdrant on ann-benchmarks datasets (sift-1m); add scalar quantization, re-measure.
• Capstone M14: vector index on node properties + distance kernels (the reference’s vec_distance.rs territory).

15. Replication, Consensus & Distribution

Why: From single node to system. Raft is table stakes; the interesting part is what each DB does differently.

• Concepts: replication topologies (leader/follower, async vs sync), redis/valkey replication + failover, Raft (leader election, log replication, snapshots, membership), consistency models (linearizability → eventual), sharding (hash slots vs ranges).
• Read code: valkey replication.c + cluster, qdrant raft-based consensus (consensus/), openraft or tikv/raft-rs, surrealdb+tikv layering.
• Papers: “In Search of an Understandable Consensus Algorithm” (Raft, ATC’14), “ZooKeeper” or “Viewstamped Replication Revisited” (for contrast), Kleppmann DDIA ch. 5, 8, 9 (read thoroughly).
• Build & bench: implement Raft leader election + log replication (or work through the raft-rs / talent-plan labs); inject partitions and observe; measure replication-lag impact of fsync policies.
• Capstone M15: ship the WAL to a follower node; then upgrade to Raft.

16. Testing & Correctness Engineering

Why: The topic that separates hobby DBs from production DBs. Turso and FoundationDB made this their identity.

• Concepts: deterministic simulation testing (DST), fault injection, property-based testing (proptest), fuzzing (cargo-fuzz/AFL), metamorphic testing (SQLancer’s pivoted queries / TLP), Jepsen & elle (checking linearizability), model checking with TLA+ (taste of), SMT solvers (Z3): proving query rewrites equivalent (Cosette-style), checking optimizer rules and constraint/invariant satisfiability.
• Read code: turso’s simulator + DST setup (they blog about it), FoundationDB simulation docs, SQLancer, antithesis blog posts, redis test/ harness, Z3 (z3.rs bindings; skim the tactic/solver architecture — treat Z3 itself as a masterclass codebase: it’s a high-performance search engine over logic).
• Papers: “Testing Database Engines via Pivoted Query Synthesis” (OSDI’20), “Finding Logic Bugs via TLP” (OOPSLA’20), Jepsen analyses (pick redis-raft and a graph DB one), “Z3: An Efficient SMT Solver” (TACAS’08), “Cosette: An Automated Prover for SQL” (CIDR’17).
• Build & bench: add proptest model-checking to the capstone (graph ops vs an in-memory model oracle); build a mini DST harness (simulated clock + fault-injecting IO layer); fuzz your parsers (Cypher + page/SST decoders); use Z3 to verify two of your topic-10 rewrite rules are equivalent (and to find a counterexample when you break one on purpose).
• Capstone M16: openCypher TCK subset runner as the correctness oracle + DST harness + fuzzers (the reference’s fuzz/ and tck_done.txt show the bar). Graduation of the correctness spine.

17. SIMD & Hardware-Conscious Data Processing

Why: The last 10x on a single core. Touched in topic 11 — this is the dedicated deep dive: writing kernels that saturate the CPU.

• Concepts: SIMD fundamentals (AVX2/AVX-512 vs ARM NEON/SVE — know both, you’re on ARM), autovectorization and why it fails, Rust portable SIMD (std::simd) vs intrinsics, branchless selection (masks + compress), SIMD hash probing (SwissTable), SIMD string parsing/comparison, bit-packed decoding at SIMD speed (FastLanes), gather/scatter costs, instruction-level parallelism & dependency chains, Mojo’s SIMD-first design (SIMD[type, width] as a first-class parametric type — compare its ergonomics vs std::simd and intrinsics).
• Read code: polars crates/polars-compute/ kernels, simdjson (the masterclass — read with the paper), hashbrown SIMD group probing, DuckDB compressed-scan kernels, usearch/SimSIMD distance functions, memchr crate, Mojo stdlib + Modular’s matmul optimization blog series.
• Papers: “Rethinking SIMD Vectorization for In-Memory Databases” (SIGMOD’15), “Parsing Gigabytes of JSON per Second” (simdjson, VLDB’19), “The FastLanes Compression Layout” (VLDB’23).
• Build & bench: write filter-selection and dot-product kernels four ways: naive scalar, autovectorized, std::simd, NEON intrinsics; bench with perf stat (IPC, vector-lane utilization); then SIMD-ize a bit-packing decoder and compare against topic 12’s scalar version; port one kernel to Mojo and compare both the numbers and the code you had to write.
• Capstone M17: SIMD-accelerated kernels in the vectorized runtime + vector-distance functions; keep scalar fallbacks and a bench comparing them.

18. GPU Acceleration for Databases

Why: GPUs are reshaping analytics, graph algorithms, and vector search — directly relevant to FalkorDB’s future (GraphBLAS on GPU exists). Learn when the PCIe tax is worth paying.

• Concepts: GPU architecture for DB people (SIMT, warps, occupancy, memory coalescing, shared memory), the data-transfer bottleneck (PCIe vs NVLink vs unified memory on Apple Silicon), GPU hash joins & aggregation, GPU graph processing (Gunrock, cuGraph, GraphBLAST — SpMV on GPU!), GPU vector search (Faiss GPU, cuVS/CAGRA), programming models: CUDA vs Metal vs wgpu/WebGPU (portable, works on your Mac) vs Mojo/MLIR (one language targeting CPU SIMD and GPU — the portability bet worth understanding).
• Read code: cuVS/RAFT (vector search kernels), libcudf (GPU columnar ops), Gunrock or GraphBLAST (graph frontier expansion), HeavyDB query compilation to GPU, Rust: wgpu compute examples, cudarc.
• Papers: “A Study of the Fundamental Performance Characteristics of GPUs and CPUs for Database Analytics” (Crystal, SIGMOD’20), “Billion-scale similarity search with GPUs” (Faiss, arXiv:1702.08734), “Gunrock” (PPoPP’16), “CAGRA: Highly Parallel Graph Construction for GPU ANN” (ICDE’24).
• Build & bench: implement filter+aggregate and batch vector-distance as wgpu compute shaders (runs on Apple Silicon Metal); bench vs your topic-17 SIMD kernels including transfer time — find the crossover batch size where GPU wins; run BFS via SpMV on GPU vs SuiteSparse CPU.
• Capstone M18: experimental GPU backend for one hot path (SpMV traversal or vector distance scoring) behind a feature flag, with CPU-vs-GPU crossover benchmark.

19. JIT & Query Compilation

Why: The other answer to interpretation overhead (vs vectorization, topic 11). HyPer/Umbra made it famous; SQLite has quietly used a bytecode VM forever; SuiteSparse:GraphBLAS JIT-compiles kernels.

• Concepts: interpreter → bytecode VM → native JIT spectrum, SQLite’s VDBE, produce/consume compilation model (HyPer), compilation latency vs execution speed (why Umbra built its own IR — “Tidy Tuples”), copy-and-patch compilation, adaptive execution (start interpreting, JIT when hot), LLVM vs cranelift vs hand-rolled backends, expression JIT vs whole-pipeline JIT, postgres’s LLVM JIT (and why it’s often a regression).
• Read code: SQLite vdbe.c (bytecode design), postgres src/backend/jit/llvm/, cranelift-jit examples, SuiteSparse:GraphBLAS JIT kernel generation (Source/jit*), DuckDB’s absence of a JIT (find the discussions — vectorization as the counter-argument).
• Papers: “Efficiently Compiling Efficient Query Plans for Modern Hardware” (Neumann, VLDB’11 — the paper), “Tidy Tuples and Flying Start” (Umbra, VLDBJ’21), “Copy-and-Patch Compilation” (OOPSLA’21), “Adaptive Execution of Compiled Queries” (ICDE’18), “Everything You Always Wanted to Know About Compiled and Vectorized Queries” (VLDB’18 — re-read after topic 11).
• Build & bench: JIT-compile filter expressions with cranelift; three-way bench: AST-walking interpreter vs vectorized (topic 11 kernel) vs JIT — including compile time; find the query length/selectivity crossover where each wins.
• Capstone M19: cranelift JIT for Cypher expressions (vs the eval.rs-style interpreter) with fallback and a compile-time budget heuristic.

20. Sparse Linear Algebra & GraphBLAS Internals (Deep Home Turf)

Why: You use the GraphBLAS API daily in FalkorDB — this topic is about owning what’s underneath: the kernels, formats, and scheduling decisions SuiteSparse makes for you.

• Concepts: sparse formats and when SuiteSparse switches between them (CSR/CSC, bitmap, full, hypersparse), SpMV vs SpMSpV, SpGEMM algorithms (Gustavson, hash-based, heap-based), masks/accumulators/semirings as an execution model, push vs pull BFS = SpMV vs masked SpMSpV (direction-optimizing), non-blocking mode & lazy evaluation, FalkorDB’s delta-matrix pattern, JIT’d kernels (ties to topic 19), GPU GraphBLAS (ties to topic 18), how SuiteSparse parallelizes: OpenMP (saxpy3’s coarse/fine task scheduling, #pragma omp parallel for static vs dynamic/guided loops, nthreads heuristics from flop counts) and the Rust alternatives: rayon work-stealing vs OpenMP static scheduling (irregular nnz-per-row is exactly where the difference shows), std::thread::scope, morsel-driven scheduling built by hand (topic 11) — note there is no mature native-Rust GraphBLAS: the crates (rustgraphblas, graphblas_sparse_linear_algebra) are FFI bindings to SuiteSparse, so a Rust rebuild must bring its own parallel runtime.
• Read code: SuiteSparse:GraphBLAS internals — format-switch heuristics, GB_AxB_* SpGEMM variants, mask handling, the OpenMP scheduling in GB_AxB_saxpy3 (how nthreads/ntasks are derived from the flopcount pre-pass); LAGraph algorithm implementations (BFS, triangle counting, PageRank); FalkorDB’s own delta-matrix layer with fresh eyes; rayon internals (join/scope, work-stealing deques) as the OpenMP counterpart.
• Papers: Davis “Algorithm 1000: SuiteSparse:GraphBLAS” (TOMS’19) + the v2 update (TOMS’23), Gustavson ’78 (two-pointer SpGEMM), Buluç & Gilbert SpGEMM survey, Beamer “Direction-Optimizing BFS” (SC’12), GraphBLAS C API spec (read cover to cover once).
• Build & bench: implement CSR SpMV and Gustavson SpGEMM in Rust; parallelize both with rayon and bench scaling 1→N cores against SuiteSparse’s OpenMP on the same matrices (SuiteSparse Matrix Collection) — measure where work-stealing beats/loses to static row partitioning on skewed (RMAT) vs uniform matrices; implement direction-optimizing BFS with masks; measure where hypersparse representation pays off.
• Capstone M20: the heart: your own sparse-matrix/GraphBLAS-subset kernels + delta matrices replace the M13 adjacency-list core; parallelism via rayon (document the OpenMP→rayon mapping decisions); benchmark both on LDBC queries, and against the reference’s graph/src/graph/graphblas layer.

21. Formal Methods & Verification

Why: Testing (topic 16) finds bugs you imagined; formal methods find the ones you didn’t. AWS, MongoDB, and CockroachDB all spec their protocols in TLA+. Also: e-graphs are quietly powering modern query optimizers.

• Concepts: SAT → SMT (DPLL(T), theories), Z3’s architecture (tactics, e-matching, the congruence closure e-graph), TLA+ & PlusCal (specify, then let TLC model-check), safety vs liveness, refinement, equality saturation with e-graphs (egg) for rewrite-rule optimizers, lightweight formal methods (spec only the scary parts), protocol testing languages (P, Ivy) as a lighter alternative, theorem proving with Lean 4 (proofs vs model checking — and Lean’s runtime itself: Perceus reference counting, functional-but-in-place updates, a systems-performance story in its own right).
• Read code: Z3 internals (src/smt/, the e-graph — a high-performance search engine over logic), egg (Rust equality saturation — read fully, it’s small), published TLA+ specs: Raft (Ongaro’s), MongoDB replication, CockroachDB’s specs repo, Lean 4 (leanprover/lean4 — the compiler/runtime in src/runtime/, and how mathlib scales proof search).
• Papers: “How Amazon Web Services Uses Formal Methods” (CACM’15 — the motivation paper), “egg: Fast and Extensible Equality Saturation” (POPL’21), “Z3: An Efficient SMT Solver” (TACAS’08), Lamport’s “Specifying Systems” (part I) + the TLA+ video course, “Cosette” (CIDR’17 — revisit from topic 16), “Counting Immutable Beans” + “Perceus: Garbage-Free Reference Counting” (the Lean/Koka runtime papers).
• Build & bench: write a TLA+ spec of the capstone’s WAL-replication protocol (topic 15) and model-check it — then remove an ack and watch TLC find the data-loss trace; build an expression-rewrite pass with egg and compare plans vs your hand-ordered rules from topic 10; in Lean 4, formalize and prove a small invariant (e.g., your B+tree ordering property or a delta-matrix merge property) — taste the proof-vs-test trade-off.
• Capstone M21: TLA+ spec of the MVCC visibility rules (or replication) checked by TLC in CI; Lean proof of a delta-matrix invariant; optional egg-based rewrite stage in the planner.

22. Standard Benchmarks: TPC-H, TPC-C, YCSB, LDBC & Friends

Why: The industry’s shared yardsticks — and their hidden messages. Knowing what each query actually stresses turns benchmarks from marketing into engineering tools.

• Concepts: OLTP vs OLAP benchmark design, TPC-C (contention, think times, and why nobody runs it honestly), TPC-H choke-point analysis (which of the 22 queries stress joins vs aggregation vs expression eval), TPC-DS, Join Order Benchmark (JOB — real data, real cardinality pain), SSB, YCSB workloads A–F & Zipfian skew, LDBC SNB + Graphalytics (graph), ann-benchmarks (vector), ClickBench, fair-benchmarking methodology & benchmarketing sins, scale factors and data generators.
• Read code/run: DuckDB’s built-in TPC-H/TPC-DS extensions, BenchBase (CMU), HammerDB, dbgen/dsdgen, LDBC SNB datagen + driver, go-ycsb/memtier.
• Papers: “TPC-H Analyzed: Hidden Messages and Lessons Learned” (Boncz — the choke-point paper, read alongside running it), “Fair Benchmarking Considered Difficult” (DBTest’18), “OLTP-Bench” (VLDB’13), “How Good Are Query Optimizers, Really?” (VLDB’15 — the JOB paper, revisit), LDBC SNB paper.
• Build & bench: run TPC-H SF10 on DuckDB and postgres, profile three choke-point queries and explain the gap; run YCSB against redis and your topic-7 RESP server; run LDBC SNB interactive on FalkorDB vs neo4j and analyze where each wins.
• Capstone M22: standing benchmark suite — LDBC SNB interactive, graph micro-benches, ann-benchmarks recall/QPS — with regression tracking across milestones, and a three-way shootout: falkordb-scratch vs falkordb-rs-next-gen vs FalkorDB.

23. Full-Text Search & Inverted Indexes (Elasticsearch / Lucene / tantivy)

Why: The third great index family after trees and hash tables. Lucene is a 25-year masterclass, tantivy is its readable Rust rival, and RediSearch is home turf.

• Concepts: inverted index anatomy (term dictionary, posting lists), text analysis pipelines (tokenizers, stemming), posting-list compression (varint, bit-packing, roaring bitmaps), FSTs for term dictionaries, BM25 scoring, top-k retrieval with WAND / block-max WAND, Lucene’s LSM-like segment architecture + merge policies (compare with topic 4!), doc values (Lucene’s columnar side), Elasticsearch distribution layer (shards, scatter-gather, relevance vs recall), hybrid search (BM25 + vectors, reciprocal rank fusion — ties to topic 14).
• Read code: tantivy (Rust, the best read — postings, FST dictionary, block-max WAND), Lucene core (codecs/, segment merging), RediSearch (redis-module perspective you know), quickwit (tantivy over object storage), Elasticsearch mostly at the architecture-docs level.
• Papers: “Inverted Files for Text Search Engines” (Zobel & Moffat, CSUR’06 — the survey), BM25 origins (Robertson & Zaragoza “The Probabilistic Relevance Framework”), “Faster Top-k Document Retrieval Using Block-Max Indexes” (SIGIR’11), “Roaring Bitmaps” (arXiv:1603.06549).
• Build & bench: build a mini inverted index in Rust: tokenize → posting lists → BM25 → top-k with block-max WAND; bench vs tantivy on a Wikipedia dump; compare roaring vs raw-vec posting lists for AND/OR queries.
• Capstone M23: full-text index on node/edge properties + hybrid search fusing BM25 with the M14 vector index (RRF) — what FalkorDB delegates to RediSearch, built in.

24. Advanced Graph Algorithms & Analytics

Why: Traversal (topic 13/20) is table stakes; the value is in analytics — centrality, communities, components — and in knowing when the algebraic (LAGraph) formulation beats the frontier-based one.

• Concepts: SSSP (delta-stepping), betweenness centrality (Brandes; batched algebraic variant), PageRank (and convergence tricks), connected components (label propagation, Afforest), community detection (Louvain → Leiden, and why Louvain’s communities can be broken), triangle counting & k-truss (masked SpGEMM!), push vs pull direction switching (Ligra), algebraic vs frontier formulations trade-offs, the GAP benchmark suite as the yardstick.
• Read code: LAGraph (the algorithm collection over GraphBLAS — study how each algorithm maps to masks/semirings; you have lagraph_lib in the reference repo already), GAP benchmark reference implementations, Ligra.
• Papers: “A Faster Algorithm for Betweenness Centrality” (Brandes ’01), “From Louvain to Leiden” (Sci. Reports ’19), “Ligra: A Lightweight Graph Processing Framework” (PPoPP’13), “The GAP Benchmark Suite” (arXiv:1508.03619), “Delta-Stepping” (Meyer & Sanders), Azad & Buluç masked-SpGEMM triangle counting.
• Build & bench: implement Brandes betweenness and Leiden in Rust over your M20 sparse core; compare against LAGraph on the same matrices (note LAGraph’s parallelism is also OpenMP — your rayon-based kernels from topic 20 carry over here); run the GAP suite (BFS, SSSP, PR, CC, BC, TC) and profile where the algebraic formulation wins/loses vs frontier-based.
• Capstone M24: LAGraph-style algorithm library over the sparse core, exposed as Cypher procedures (CALL algo.pagerank(...) — FalkorDB-style).

25. Graph Neural Networks & Graph ML

Why: Message passing is SpMM over a semiring — your GraphBLAS core is already a GNN engine waiting to happen. And GraphRAG (which you know from GraphRAG-SDK) is pulling graph DBs into the ML serving path.

• Concepts: node embeddings (DeepWalk, node2vec — random walks + skip-gram), message passing as generalized SpMM, GCN / GraphSAGE / GAT (and what each adds), mini-batch neighbor sampling for graphs that don’t fit (GraphSAGE’s real contribution), knowledge-graph embeddings (TransE family), GNN systems view: how PyG/DGL kernels map to sparse ops, embeddings-in-the-database (compute → store in vector index → hybrid query), GraphRAG architectures.
• Read code: DGL / PyTorch Geometric sparse kernels (the SpMM/SDDMM ops underneath), candle or burn (Rust ML — for implementing), your own GraphRAG-SDK with fresh systems eyes.
• Papers: “node2vec” (KDD’16), “Semi-Supervised Classification with GCNs” (Kipf & Welling, ICLR’17), “Inductive Representation Learning on Large Graphs” (GraphSAGE, NeurIPS’17), “Graph Attention Networks” (ICLR’18), “TransE” (NeurIPS’13), “Graph Neural Networks meet Databases” survey (pick a recent arXiv one when starting).
• Build & bench: implement node2vec and a 2-layer GCN in Rust (candle/burn) using your own M20 SpMM as the aggregation kernel; train on Cora and ogbn-arxiv; bench your SpMM against DGL’s on the same graphs; store the learned embeddings in your M14 vector index and measure end-to-end hybrid query latency.
• Capstone M25: embeddings pipeline — compute node2vec/GCN embeddings with your own kernels, store them in the vector index, and answer GraphRAG-style hybrid queries (pattern match + semantic similarity) in one Cypher query.

26. Indexing & Probabilistic Data Structures

Why: Indexes are bets — you pay write amplification for read speed. And the probabilistic structures (bloom filters, HLL — redis’s PFCOUNT is one) buy huge wins by being slightly wrong.

• Concepts: secondary index design and its write cost, composite/covering indexes & index-only scans, hash vs B-tree vs bitmap vs BRIN (≈ zone maps), partial & expression indexes, index maintenance under MVCC (postgres HOT, index bloat), index selection (“what-if” analysis), learned indexes (RMI, ALEX, PGM — do they survive contact with updates?); spatial/geo indexes: R-tree & R*-tree (bounding-box hierarchy, node splits), quadtrees & kd-trees, space-filling curves that turn 2-D into a 1-D B-tree problem (Z-order/geohash, Hilbert — and why Hilbert clusters better), S2/H3 cell coverings, redis GEO (a 52-bit geohash stuffed into a zset — indexes-you-already-have reuse), postgres GiST/SP-GiST as the extensible index framework spatial rides on (nearest-neighbor via priority-queue traversal); compressed bitmaps: roaring internals (array/bitmap/run containers, galloping intersection), WAH/EWAH ancestry, SIMD-accelerated set operations, where they power real systems (Lucene doc sets, ClickHouse, Druid, Pilosa); succinct structures (rank/select, Elias-Fano encoding of sorted IDs — postings and adjacency lists both); probabilistic filters: bloom filter math (FPR vs bits/key), blocked bloom (cache-line friendly), cuckoo, xor, ribbon filters (RocksDB’s evolution); sketches: HyperLogLog (dense/sparse), count-min, t-digest, top-k.
• Read code: postgres index access methods (nbtree/, gin/, brin/, gist/ + PostGIS’s R-tree-over-GiST), redis geo.c/geohash.c (the zset trick end-to-end), s2geometry or h3 (cell covering APIs), RocksDB util/bloom* + ribbon filter, redis hyperloglog.c (the dense/sparse encoding dance — a classic), RedisBloom module, CRoaring + roaring-rs (container switching, SIMD intersections), Lucene RoaringDocIdSet, PGM-index and ALEX repos.
• Papers: “The Case for Learned Index Structures” (SIGMOD’18), “ALEX” (SIGMOD’20), “The PGM-index” (VLDB’20), “R-trees: A Dynamic Index Structure for Spatial Searching” (Guttman, SIGMOD’84), “The R*-tree” (SIGMOD’90), “Better bitmap performance with Roaring bitmaps” (SPE’16) + “Roaring Bitmaps: Implementation of an Optimized Software Library” (SPE’18), “Cuckoo Filter: Practically Better Than Bloom” (CoNEXT’14), “Xor Filters” (JEA’20), “Ribbon Filter” (arXiv:2103.02515), “HyperLogLog in Practice” (Google, EDBT’13).
• Build & bench: implement a mini roaring bitmap (three container types + adaptive switching) and bench intersect/union vs roaring-rs and a plain HashSet<u32> across densities — find where each container wins; implement blocked-bloom, cuckoo, and xor filters — bench FPR vs bits-per-key vs lookup latency in one chart; implement HLL and verify the error bound empirically; implement a Z-order/geohash index over your M3 B+tree plus a small in-memory R-tree — bench point-in-radius and bounding-box queries vs full scan across selectivities, and measure where the curve’s “cell boundary” false positives hurt; race a PGM-index against your M3 B+tree, then add updates and watch the story change.
• Capstone M26: secondary range indexes maintained under MVCC + bloom filters in the LSM backend + roaring bitmaps for label/type filtering in pattern matching + HLL fast path for approximate count(DISTINCT ...) in Cypher + geo index for point properties (Z-order over the range index) answering WHERE distance(n.loc, $p) < r — FalkorDB has a point type; make it indexable.

27. Streaming & Incremental View Maintenance

Why: Recomputing from scratch is the enemy. Differential dataflow and DBSP made incremental computation rigorous — and FalkorDB’s delta matrices are already halfway there conceptually.

• Concepts: dataflow model (timely), differential dataflow (deltas all the way down), DBSP (the algebraic theory of incremental computation — Z-sets will feel familiar after semirings), materialized view maintenance, watermarks & out-of-order data, exactly-once semantics, the log as the database (Kafka), incremental graph queries (registered/standing Cypher queries).
• Read code: differential-dataflow + timely (Rust, Frank McSherry), Feldera (DBSP implementation, Rust), Materialize architecture, RisingWave (Rust streaming DB).
• Papers: “Naiad: A Timely Dataflow System” (SOSP’13), “Differential Dataflow” (CIDR’13), “DBSP: Automatic Incremental View Maintenance for Rich Query Languages” (VLDB’23 best paper), “Kafka” (NetDB’11).
• Build & bench: incremental PageRank and triangle counting with differential-dataflow — stream edge insertions and compare incremental-update cost vs full recompute as the graph grows; write a delta-join operator by hand to demystify it.
• Capstone M27: standing Cypher queries — register a query, keep its result incrementally maintained under graph mutations via delta matrices, push changes to subscribers.

28. Cloud-Native & Disaggregated Storage

Why: The architecture every serious DB is converging on: compute is stateless, the log/object store is the database. Aurora, Neon, Snowflake — and it changes every design trade-off you learned in topics 3–6.

• Concepts: compute–storage separation, Aurora’s “the log is the database”, Neon’s pageserver + WAL-redo model, object storage as substrate (S3 latency/cost/consistency model), caching tiers & request hedging, snapshots and copy-on-write branching, serverless & scale-to-zero, shared-data vs shared-nothing, LSM tiering to object storage.
• Read code: neon (Rust — pageserver, safekeepers), slatedb (Rust LSM on object storage — small and current), quickwit (search over S3), turso’s object-store work.
• Papers: “Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases” (SIGMOD’17), “Socrates: The New SQL Server in the Cloud” (SIGMOD’19), “The Snowflake Elastic Data Warehouse” (SIGMOD’16), “Building a Database on S3” (SIGMOD’08 — prescient), Neon architecture posts.
• Build & bench: move your LSM backend’s SSTs to object storage (MinIO locally) with a local NVMe cache tier; measure p50/p99 read latencies vs local-only and tune the cache; implement copy-on-write graph branching (Neon-style branches for graphs).
• Capstone M28: tiered storage backend — hot data local, SSTs on object storage — plus instant graph snapshots/branches.

29. Distributed Transactions

Why: The layer above topic 15’s Raft: making transactions span shards. The gap between 2PC-in-a-textbook and Spanner/FoundationDB is where the deep understanding lives.

• Concepts: 2PC and its blocking failure mode, Percolator (transactions over a KV store — TiKV’s model), Spanner’s TrueTime + external consistency, hybrid logical clocks (HLC — CockroachDB’s answer to no atomic clocks), Calvin & deterministic databases (Abadi’s counterpoint), FoundationDB’s decomposed architecture (sequencer/resolvers/storage), contention & abort-rate dynamics, the cross-shard graph traversal problem (why graph partitioning is hard).
• Read code: tikv (txn/ — Percolator in Rust), FoundationDB (with the SIGMOD’21 paper as the map), CockroachDB kv/txn coordinator + HLC.
• Papers: “Spanner” (OSDI’12), “Large-scale Incremental Processing Using Distributed Transactions” (Percolator, OSDI’10), “Calvin” (SIGMOD’12), “FoundationDB: A Distributed Unbundled Transactional Key Value Store” (SIGMOD’21), “Logical Physical Clocks” (HLC, OPODIS’14).
• Build & bench: shard your graph across two processes; implement 2PC, then Percolator-style transactions over the KV layer; drive both with the M16 DST harness injecting crashes at every 2PC state; measure abort rates vs contention (Zipfian hot keys).
• Capstone M29: cross-shard transactions + cross-shard pattern matching over a partitioned graph.

30. Time-Series Engines

Why: Small, beautiful, and immediately useful: Gorilla’s encodings are the best compression-ratio-per-line-of-code in databases. And temporal graphs are an open frontier for FalkorDB.

• Concepts: Gorilla compression (delta-of-delta timestamps, XOR floats), time-partitioned storage & retention/downsampling, out-of-order ingestion (the hard part), tag inverted index (series lookup — topic 23 reappears), high-cardinality pain, IOx architecture (DataFusion + Parquet + object storage — topics 11/12/28 combined), TSBS benchmarking.
• Read code: influxdb (IOx engine, Rust), prometheus tsdb/ (Go, very readable — head block + WAL + compaction), VictoriaMetrics (ruthless efficiency).
• Papers: “Gorilla: A Fast, Scalable, In-Memory Time Series Database” (VLDB’15 — read first), “Monarch: Google’s Planet-Scale In-Memory Time Series Database” (VLDB’20), “BtrDB” (FAST’16).
• Build & bench: implement the Gorilla codec (delta-of-delta + XOR floats) in Rust; bench compression ratio and decode throughput on real metrics (node_exporter dumps) vs Parquet+zstd; handle out-of-order writes and measure the cost.
• Capstone M30: temporal graph support — edge/property history with Gorilla-compressed values and time-travel pattern matching (MATCH ... AT TIME t).

31. CRDTs & Multi-Master Replication

Why: The anti-consensus: let replicas diverge and merge deterministically. Redis Enterprise’s active-active CRDB is built on this — an active-active graph is a genuinely hard, genuinely interesting design problem.

• Concepts: strong eventual consistency, state-based vs op-based CRDTs, the classics (G-Counter, PN-Counter, LWW-Register, OR-Set), causality tracking (vector clocks, dots), sequence CRDTs (RGA, Fugue — why collaborative text is the hard case), JSON/tree CRDTs and the move-operation problem, when CRDTs beat consensus and when they quietly lose data (LWW’s lie), local-first software, graph CRDTs: OR-Set nodes/edges + LWW property maps, and the dangling-edge problem.
• Read code: automerge (Rust), loro (Rust — fast, modern engine), yrs (Yjs port), cr-sqlite (CRDT layer bolted onto SQLite — instructive architecture), diamond-types.
• Papers: “Conflict-free Replicated Data Types” (Shapiro et al., SSS’11 — the founding paper + the INRIA comprehensive study), “A Conflict-Free Replicated JSON Datatype” (Kleppmann & Beresford ’17), “A Highly-Available Move Operation for Replicated Trees” (Kleppmann ’21), “Local-First Software” (Onward! ’19), Loro/Fugue blog series on sequence CRDT performance.
• Build & bench: implement PN-Counter and OR-Set, property-test convergence (proptest: any permutation of concurrent ops merges to the same state — a beautiful proptest target); bench automerge vs loro on the editing-trace benchmarks; design a graph CRDT on paper first: what happens to an edge when one replica deletes its endpoint?
• Capstone M31: active-active mode — two masters accepting writes, OR-Set nodes/edges + LWW properties, deterministic merge; contrast its guarantees and latency with the M15 Raft path on the same workload.

32. HTAP Architectures

Why: Every earlier topic picked a side — row/OLTP (3, 8) or column/OLAP (12). HTAP is the refusal to pick: transactional writes AND analytical scans on the same (logical) data. FalkorDB has the same split waiting: OLTP graph mutations vs topic-24 analytics that want a stable, columnar-ish view.

• Concepts: the freshness/isolation/interference triangle (the HTAP trilemma); the architecture menu — separate copies wired by replication (TiDB→TiFlash: columnar replicas as Raft learners, consistent reads via learner-read + Raft index wait), dual formats in one engine (SAP HANA delta+main, Oracle Database In-Memory dual-format, SingleStore rowstore→columnstore), snapshot-the-memory (HyPer’s fork()-based virtual-memory snapshots), lakehouse-ish decoupled (F1 Lightning: CDC into a read-optimized store); delta-main merge policies (rhymes hard with FalkorDB delta matrices AND topic 4’s LSM); planner routing — one optimizer choosing row vs columnar replica per (sub)query with a freshness bound; resource isolation so scans don’t starve p99 writes; CDC/changelog as the universal glue (topic 27’s log-is-the-database, applied).
• Read code: TiFlash (DeltaTree storage — delta layer + stable layer, the merge), TiDB planner’s TiKV-vs-TiFlash cost-based routing + learner-read wait, SingleStore/HANA architecture docs (no source, read the papers/blogs as specs), DuckDB-inside-Postgres extensions (pg_duckdb/pg_analytics) as the budget HTAP pattern.
• Papers: “TiDB: A Raft-based HTAP Database” (VLDB’20 — the must-read), “Hyper: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots” (ICDE’11), “SAP HANA Database: Data Management for Modern Business Applications” (SIGMOD Record ’12), “F1 Lightning: HTAP as a Service” (VLDB’20), “Real-Time Analytics: The HTAP Survey” (Özcan et al., SIGMOD’17 tutorial).
• Build & bench: measure the interference first — run topic 22’s YCSB-style write workload concurrently with full-column scans on one engine and chart p99-write vs scan-throughput; then split: maintain a columnar replica from your WAL/changelog and re-measure both sides + the freshness lag; implement learner-read semantics (reads wait for a replication watermark) and price the wait.
• Capstone M32: HTAP FalkorDB — the M27 changelog feeds a read-optimized analytical replica (columnar property store from M12 + stable GraphBLAS matrices without delta overlays from M20); route topic-24 CALL algo.* and heavy aggregations to it with a declared freshness bound (AS OF watermark), keep OLTP mutations on the primary; bench interference eliminated vs the single-copy engine, TiDB-style.

After the plan (ideas backlog)

• FPGA / SmartNIC / computational storage offload (beyond GPU)

Progress

Status: todo → in progress → done. Add a one-line takeaway when done.

Capstone milestones (falkordb-rs-next-gen from scratch)

Session log

• 2026-07-12 — restructure rollout: all 179 remaining reading guides across topics 00-25 and 27-32 rewritten as self-contained chapters (topic 26 was the pilot, previous entry), executed by 8 parallel agents (4 topics each) against the same spec: concept-first H1 titles replacing “Reading guide — …”, Sources blocks replaced by 2-4 sentence framing leads, one inline Rust-ish code sample of the core algorithm added where the guide lacked one (skips documented for pure surveys / guides already carrying equivalent code), all existing content kept (diagrams, line-anchor tables, questions, tie-backs), ## References appended with Papers (arXiv) + Code (GitHub) links carrying the old reading advice, filenames unchanged to avoid link churn; the 32 topic READMEs’ guide lists updated to the new titles; one agent (topics 16-19) hit context limits with 2 files left — reading-umbra-tidy-tuples.md (retitled “Umbra & copy-and-patch: the war on compile latency” + copy-and-patch memcpy/patch-holes sample + References) and reading-sqlite-vdbe.md (References) finished by hand; SUMMARY.md link titles regenerated centrally by script from the actual on-disk H1s rather than agent reports (179 updated); verification: zero old-style H1s, zero Sources blocks, zero guides missing References, fence-and-backtick-aware bare-angle-bracket scan across all 186 guides found one genuine hazard (HashMap<fd, handler> in topic 7’s ae guide, backticked), mdBook build green. The whole book now reads as chapters instead of pointers.
• 2026-07-12 — book-quality pass, three moves: (1) paper audit against dbscholar’s citation-PageRank ranking (rmarcus.info — pulled the underlying data.json, 11,867 SIGMOD/VLDB/CIDR/PODS papers) — resources/papers.md gains a “Modern systems & directions” section and 6 topic READMEs gain “Further references” (Kung-Robinson OCC ’81, Calcite, Spark SQL, Kipf/Neo/Bao learned optimization, Photon, Velox, GAMMA, Dremel, Lakehouse+Delta Lake, MillWheel, CockroachDB); (2) all ~/repos/... code references linkified to their GitHub repos (scripted, fence-aware: 143 links across 115 files) so the online book’s code pointers resolve; (3) restructure demo on topic 26 — all 7 reading guides rewritten as self-contained chapters: concept titles (“HyperLogLog: count distinct in 12 KB” not “Reading guide — …”), framing lead instead of a Sources block, an inline Rust code sample of each core algorithm (HLL add/merge + Ertl count skeleton, blocked-bloom 6-probe loop with golden-ratio remix, cuckoo kick loop with the XOR involution, PGM shrinking-cone add_point, roaring galloping intersect, BRIN one-sided range prune, Morton interleave64 magic masks), and a “## References” section at the bottom (papers with arXiv links, code with GitHub links); filenames kept (reading-*.md) to avoid link churn; SUMMARY.md + README titles updated; mdBook build verified locally (mdbook-mermaid install + build green). If the format lands, roll it out to the other 32 topics.
• 2026-07-12 — PLAN.md expansions backfilled into the four already-scaffolded topics (the plan-only commit 2fb095a now has matching study material): topic 7 gains “Bolt: the third answer” — RESP/pgwire/Bolt framing-typing-streaming table (Bolt’s PULL{n}/DISCARD = protocol-level backpressure, §4’s problem solved at the wire) + reading-bolt-packstream.md anchored on FalkorDB’s removed Bolt server read frozen via git show 0b11a00b3^:src/bolt/ (#2170, 2026-07-08): session-sequence ASCII with handshake bolt_api.c:803/version-clamp :845-864, RUN-executes-but-PULL-streams :467-482/:504-521 decoupling, PackStream marker nibbles bolt.c:11/:21/:36 + graph types Node 0x4E/Rel 0x52/Path 0x50 in the type system, why-was-it-removed as a question, M7 stretch = Bolt listener beside RESP; topic 13 gains “The query-language landscape” — 6-language table (Cypher/GQL/SQL-PGQ/SPARQL/Gremlin/Datalog) on model/matching/composability/pushdown + reading-query-languages.md (SIGMOD ’22 GQL+PGQ paper, count-2-paths-in-a-triangle under homomorphism/isomorphism/trail as the same-pattern-three-answers demo, family-tree mermaid, kuzu Cypher.g4 anchor, M13 rule: keep the AST GQL-shaped — quantified path patterns + explicit path-mode field); topic 20 gains “Parallelism: OpenMP inside SuiteSparse, rayon in Rust” — saxpy3’s costed static scheduling (coarse/fine tasks GB_AxB_saxpy3.c:22-48, nthreads-from-flopcount slice_balanced.c:418, parallel flopcount pass :219) vs rayon work-stealing (join/mod.rs:93 inline+push+steal, registry.rs:248 crossbeam_deque Stealer) + reading-openmp-vs-rayon.md with the static-vs-stealing trade table, no-native-Rust-GraphBLAS note (crates are FFI), new M20 checkbox: document each OpenMP→rayon mapping decision; topic 26 gains “Geo indexes: 2D keys through 1D indexes” — valkey GEO as geohash-in-a-zset (interleave64 geohash.c:52 → 52-bit Morton score, geohashEstimateStepsByRadius helper.c:64, 9-cell scan geo.c:375 + haversine verify = bloom’s candidate-then-verify control flow) + reading-geo-indexes.md (Z-order seams vs Hilbert, Guttman R-tree/GiST, S2 prefix-containment vs H3 hexagons, M26 mapping: Morton key through the existing sorted property index). SUMMARY.md gains the 4 new guides so they appear in the book.
• 2026-07-11 — topic 32 scaffolded (added to the plan this session, so the scaffold-all-topics run is complete again): study guide (the HTAP problem measured — bench lane 1 run: 1M-row store behind one coarse lock, fixed 2 s window per mode: writes alone = 11,438,647 writes at p99 333 ns; writes + a free-running full scanner = 69 writes with p99 7.49 seconds — not slowdown, starvation: std Mutex is unfair, the scanner re-wins the lock after each ~0.6 ms scan (3261 scans) and the parked writer never gets in; interference at its worst is zero writes, and the coarse lock is deliberate — mitigations ARE the topic; the freshness/isolation/cost trilemma triangle; the architecture-menu table HANA-delta+main / HyPer-fork() / TiFlash-learner / F1-Lightning-CDC / pg_duckdb-offload keyed on one-copy? freshness isolation; the changelog-is-the-glue mermaid tying topic 27’s thesis to every split design; the same-fold-four-costumes thread: topic 4 LSM minor compaction = HANA delta merge = TiFlash segmentMergeDelta = FalkorDB delta-matrix flush), 4 reading guides (TiDB VLDB ’20 — columnar-copy-as-Raft-LEARNER so the replica costs no write-quorum latency, freshness-is-a-WAIT with tiflash LearnerRead.cpp:35 doLearnerRead + :61 waitIndexTimeout as the anchor, one-planner-two-engines via tidb find_best_task.go:535/:1841/:1878 TiFlash-paths-retained-so-cost-not-topology-decides, learner-log vs CDC-changelog trade question; TiFlash DeltaTree — Segment.h:84 delta-over-stable ASCII with MemTableSet/DeltaValueSpace.h:65 as a-little-LSM-inside-the-delta, Delta/MinorCompaction.h why-compact-the-delta-at-all, DeltaIndex.h:27 as the structure that makes delta+stable merge reads cheap (the thing our scan_sum_a lacks), DeltaMergeStore.h:668 segmentMergeDelta with our merge_preserves_scans test as its correctness condition, MVCC-versions-in-both-layers GC question linking topic 31’s causal stability; HyPer ICDE ’11 + HANA SIGMOD Record ’12 paired as the one-copy family — fork() CoW-page ASCII (snapshot cost ∝ dirtied pages = MVCC-where-the-version-chain-is-the-page-table, GC is exit()), snapshot-ages-until-re-fork = lane 3’s apply interval in OS clothing, HANA delta+main = our replica.rs minus segmenting, HANA’s-trilemma-corner-is-exactly-what-lane-1-measures; F1 Lightning VLDB ’20 + Özcan SIGMOD ’17 survey — CDC-fed HTAP with zero OLTP changes, safe-timestamp = applied_lsn productionized (reads never wait, served stale-but-consistent at the max fully-applied ts — the opposite choice from doLearnerRead), Changepump ordering question = topic 27 changelog + topic 29 Spanner timestamps, the survey’s copies×engines quadrant with every cell trading the same three currencies), experiments crate compiles: row.rs PROVIDED (RowStore = rows + every-write-appended-to-log changelog + scan oracle + skewed_key + percentile — 2 provided tests pass) — replica.rs (ColumnarReplica delta+main: apply/scan_sum_a/merge_delta with delta-overrides-main + highest-lsn-per-key-wins + merge-preserves-scans-and-sorts contracts, freshness_is_visible via applied_lsn gap — TiFlash DeltaTree in miniature, 4 tests), learner.rs (read_wait over an apply schedule: first-batch-covering-read_index, Some(0) if already applied, None = waitIndexTimeout — doLearnerRead as arithmetic, 3 tests) are todo!() stubs — 7 tests fail as todo panics; htap_bench lane 1 RUN (table above; original fixed-200K-writes design serialized into minutes behind ~0.6 ms scan lock-holds — switched to fixed-2s-window per mode, making the writes-completed collapse the headline number), lanes 2 (scan row-vs-delta-heavy-vs-merged + freshness max-lsn-gap vs batch 1K/10K/100K) and 3 (learner-read wait distribution vs apply interval 1/10/100, 50K reads demanding lsn==now) armed behind catch_unwind; notes.md predicts lanes 2-3 + records the starvation surprise + the RwLock exercise; M32 log: Lightning-shaped not TiFlash-shaped (no consensus group until M15, decoupling = zero primary changes), M27’s changelog feeds a delta-matrix replica, router advertises applied_lsn safe timestamps + freshness-bound routing with read_wait-or-fallback, before-shot recorded: 69 writes/2s when analytics shares the copy, success = restoring the 11.4M with scans elsewhere. Cloned tiflash + tidb.
• 2026-07-11 — topic 31 scaffolded (the last topic): study guide (consensus-vs-CRDT as the same problem in opposite currencies — agree-on-an-order vs design-so-order-doesn’t-matter, 1-RTT vs 0-RTT, unavailable-in-minority vs available-under-any-partition; SEC via join-semilattice mermaid; CvRDT/CmRDT table with where-each-lives-in-this-crate; the zoo table clock/lww/counter/orset/rga/graph with the one idea per structure; LWW’s lie measured — lane 1 run: two replicas, 20K writes each, LWW map: 10 hot keys + sync-every-write loses 94.98% of writes, 1000 keys + sync-every-100 loses 88.34%, even 100K keys + rare sync loses 12.45% — “eventually consistent” without conflict semantics is not a semantics, priced; sequence-CRDT integration ASCII with the interleaving dragon; code-reading table across all five cloned repos), 4 reading guides (Shapiro SSS’11 + INRIA RR-7506 — SEC’s three clauses, the CvRDT⇔CmRDT equivalence proof, the catalog reading map ending at §4’s graphs where concurrent addEdge∥removeVertex is declared application-specific = the dangling-edge problem M31 inherits, causal-stability GC question; Kleppmann arc JSON-CRDT ’17 + move-op ’21 + Local-First Onward!’19 — ops-address-identities-not-paths with automerge op_set2/op.rs:52 succ as deletion-by-successor-ops, the concurrent-move duplicate/cycle problem and its undo/redo total-order fix, the does-move-survive-in-graphs M31 design question; sequence CRDTs in production — yrs block.rs:160 ID=our-Dot / :1302 Item with origin+right_origin=YATA’s pair vs RGA’s single parent / :1415 Item::integrate as the loop our rga.rs apply implements plus run-coalescing splits, diamond-types merge.rs:142 self-described “bastardization” + yjsspan.rs:29 INSERTED/NOT_INSERTED_YET retreat/advance = only-be-a-CRDT-at-merge-time, Loro/Fugue maximal-non-interleaving with the letter-soup demo, automerge-vs-loro bench as scratch-project exercise per deps convention; cr-sqlite as THE-database-goes-multi-master — crsql_as_crr clock-row-per-CELL diagram, local_writes/mod.rs:83-133 db_version bookkeeping = Lamport-clock spine, compare_values.rs version-tie-broken-by-VALUE-comparison = deterministic convergence with zero clock trust, changes_vtab.rs replication-endpoint-as-virtual-table, delete-wins-for-rows vs our add-wins-for-nodes tension question, the M31 change-feed schema design question), experiments crate compiles: clock.rs PROVIDED (Dot + VClock tick/covers/merge/partial_cmp-None-defines-concurrent, modeled on automerge clock.rs:109/:145) + lww.rs PROVIDED (register/map with (ts,replica) total order; merge counts its own discards so lane 1 can price the lie) — 6 provided tests pass — counter.rs (G/PN with semilattice-law tests + why-PN-is-two-G-Counters), orset.rs (add-wins over Dots: add-tags-fresh-dot / remove-kills-observed-dots, concurrent-add-beats-remove + remove-covers-all-observed-tags + seeded-permutation convergence), rga.rs (insert-after-parent + skip-larger-(counter,replica)-siblings + tombstones-still-anchor, idempotent apply, delete∥insert-after-it convergence), graph.rs (OR-Set nodes/edges + LwwMap props composition; dangling-edge-hidden-NOT-deleted test: edges()-filters-to-visible-endpoints, re-add-resurrects; props keyed by node id survive remove/re-add vs automerge’s key-by-creation-op as an exercise) are todo!() stubs — 18 tests fail as todo panics; crdt_bench lane 1 RUN (table above; also caught state-based-sync’s quadratic cost live — sync_every=1 ships the whole map per write, workload shrunk and the delta-CRDT motivation written into the bench comment), lanes 2-4 armed behind catch_unwind (OR-Set gossip storm + tombstone census, RGA 50K-char trace + tombstone bloat, graph dangling storm 100-removes∥500-edge-adds + resurrection count); notes.md flags lane 1’s honest caveat (counts merge-time discards only — locally-overwritten writes don’t show, so it’s a lower bound); M31 log: node/edge identity must be Dots not user ids (cr-sqlite’s auto-increment-PK trap), dangling policy locked hide-not-delete, props LWW-with-HLC (topic 29), anti-entropy v1 whole-state → v2 db_version-watermark deltas, deliverable = same workload through M15 Raft vs active-active with latency histogram + concrete-conflict table. All 32 topics scaffolded.
• 2026-07-10 — topic 30 scaffolded: study guide (the-shape-of-the-problem ASCII — regular append-mostly writes vs range+selector+aggregation reads; the baseline finding measured: delta+varint lands at a shape-blind 11.00 B/sample because raw f64 values dominate — whatever the codec does about timestamps is a rounding error until it attacks the value bytes, hence XOR; Gorilla→Prometheus/VM→IOx lineage mermaid with Monarch and BtrDB as the bracketing extremes; five-system table on codec / time organization / label index / out-of-order policy; TSDB=LSM-keyed-by-time thesis with retention=drop-the-oldest-level), 4 reading guides (Gorilla VLDB ’15 + prometheus chunkenc/xor.go — prediction-error framing with the dod/XOR ASCII, paper bucket table vs prometheus’s retuned 14/17/20 buckets (:195-208) as buckets-are-workload-parameters, writeVDelta :226 window reuse, :396 tDelta+=dod as the whole model in one line, no-random-access-by-design, entropy-floor bit accounting question; prometheus tsdb — full architecture ASCII head/WAL/2h-blocks/exponential compaction, head_append.go:436/:481/:688-693 as the exact head.rs contract (ErrOutOfOrderSample vs ErrTooOldSample), OutOfOrderTimeWindow head.go:168 quarantine design, MemPostings postings.go:60/:403 = topic 23’s inverted index with labels as terms, the two famous failure modes high-cardinality + churn; VictoriaMetrics + InfluxDB 3 paired as two-rebuttals — VM partition.go:75 rawRows→parts LSM-said-out-loud, nearest_delta2.go:15 byte-aligned varint batches with optionally-lossy precisionBits vs Gorilla’s exact bits, index_db.go:124 tagFilters cache + churn invalidation, vs IOx WAL→Arrow QueryableBuffer→Parquet-on-object-store (influxdb3_wal/lib.rs:75-98 SnapshotTracker, queryable_buffer.rs:41) as topic 28’s landing-zone applied to metrics, vertical-integration-vs-commodity-formats trade table, how-much-of-Gorilla’s-win-was-really-sorting question; Monarch VLDB ’20 + BtrDB FAST ’16 — monitoring-must-not-depend-on-what-it-monitors ⇒ RAM-first lazy durability, push-vs-pull, distribution-typed values as the schema cure for cardinality, query pushdown; BtrDB’s aggregate tree — min/mean/max/count per 64-way node ⇒ query cost ∝ pixels not samples, downsampling as the index structure itself, CoW versions), experiments crate compiles: gen (scrape jitter, gauge/counter/constant/random shapes, OOO arrivals, label sets with the unique-instance cardinality bomb) + bits (MSB-first BitWriter/Reader + sign_extend so the stub is the algorithm not bit plumbing) + baseline (zigzag varint delta) PROVIDED — 7 provided tests pass, lane 1 RUN (table above, decode ~270-330 Msamples/s — byte-aligned codecs have a real throughput edge over bit-packed Gorilla, the actual design axis is ratio-vs-decode-speed) — gorilla.rs (paper dod buckets + XOR leading/trailing windows; bit-exact roundtrip incl. bucket edges, constant ≤~2 bits/sample, gauge beats raw 3×, random must FAIL to compress >8 B/sample — the codec wins on regularity not magic), head.rs (in-order fast path + bounded OOO window + TooOld refusal + LWW merge flush that feeds the in-order-only encoder — prometheus semantics exactly), index.rs (MemPostings-style (name,value)→sorted-ids + shortest-list-first k-way intersect, brute-force oracle, cardinality-bomb-counted test) are todo!() stubs — 15 tests fail as todo panics; tsdb_bench lanes 2-4 (gorilla ratios per shape, OOO tax sweep 0-50%, selector latency at 100K series) armed behind catch_unwind; notes.md predicts all lanes; M30 log: history chunks per (entity, attribute), MATCH ... AT TIME t = latest_write_before ≤ t so M29’s MVCC read path generalizes to time-travel, storage split by age (custom hot chunks → M28 Parquet cold), Gorilla dod survives for changelog timestamps but property values need dictionary+RLE not XOR, BtrDB-shaped rollup tree over the M27 changelog for graph-evolution queries.
• 2026-07-10 — topic 29 scaffolded: study guide (the-problem-priced motivation table — measured conflict probability of the bank workload itself: 0.3% of 8-txn batches collide at zipf θ=0.5 but 29.9% at 0.9, 86.2% at 1.1, 99.6% at 1.3 — contention is the common case at real-workload skew; design-space mermaid rooted at textbook 2PC with the three escapes labeled on the edges — move-the-decision-into-the-data (Percolator), replicate-the-coordinator (Spanner), remove-runtime-agreement (Calvin), decompose+batch (FDB); one-table five-system summary keyed on concurrency control / clock / cross-shard atomicity / blocking window; Percolator-in-six-lines with THE-COMMIT-POINT marked), 4 reading guides (Percolator OSDI ’10 + TiKV — three-column-families ASCII (data/lock/write with write-CF-as-commit-index), lifecycle sequenceDiagram with the any-reader-resolves note, TiKV walk actions/prewrite.rs:37 (pessimistic_action + secondary_keys args as post-paper hardening) → commit.rs:64 with the :57 duplicate-commit-returns-Ok idempotency arm → check_txn_status.rs:92/:241 + MissingLockAction :458 as production resolve_lock → cleanup.rs:24 Rollback records our sim skips → latch.rs/scheduler.rs local-vs-distributed conflict split, txn_status_cache; Spanner OSDI ’12 + HLC OPODIS ’14 paired — bound-the-ERROR-vs-bound-the-SKEW fork diagram, commit-wait derivation, HLC rules + the l≤max-pt anti-Lamport-drift bound, uncertainty-interval restarts, CRDB walk hlc.go:38/:411/:471/:517 (UpdateAndCheckMaxOffset crashes the node — maxOffset is a promise) + txn_coord_sender.go:113 interceptor stack + txn_interceptor_committer.go:128 parallel commits with STAGING-is-implicitly-committed :195-205 as Percolator’s-resolve-idea-shaving-a-latency-round; Calvin SIGMOD ’12 — agree-on-inputs-not-outcomes diagram, sequencer/scheduler/executor layers, deterministic-locking-kills-both-deadlock-and-2PC, OLLP reconnaissance for dependent txns with the graph-traversals-are-the-ultimate-dependent-txn M29 question; FoundationDB SIGMOD ’21 — unbundled roles ASCII, ConflictSet.cpp:947 detectConflicts over the :224 SkipList as the-whole-SI-check-in-one-data-structure, CommitBatchContext :504 batch-is-the-unit, masterserver-is-barely-a-counter, ResolverBug.cpp injectable-wrong-answers as DST culture beyond crash injection, vs-Calvin and vs-Percolator design reads), experiments crate compiles: kv.rs PROVIDED (the three Percolator column families as HashMaps/BTreeMap, strictly-monotonic TSO, latest_write_before range scan, 2-shard cluster, Zipf transfer workload — 4 provided tests pass) — tpc.rs (2PC coordinator with 4-point CrashPoint injection + recovery-from-durable-state-only; blocking_window_demonstrated test names the flaw), percolator.rs (get/prewrite/commit_primary/commit_secondaries/resolve_lock with roll-forward-iff-primary-committed tests, no-lock-leak on failed prewrite, total-conserved-after-rollback), hlc.rs (send/recv rules; monotonic-under-backward-clocks, l-bounded-by-max-pt over 1000 skewed messages, concurrent-events-collide-without-node-id-tiebreak as an assert_eq teaching test) are todo!() stubs — 14 tests fail as todo panics; txn_bench lane 1 RUN (conflict table above), lanes 2 (abort rate vs θ + bank invariant) and 3 (20K-txn crash storm, crash every 100th cycling 4 points, blocked_aborts counts the blocking window empirically) armed behind catch_unwind; notes.md predicts lane 2/3 numbers before implementation; M29 log: shard by node id, cross-shard edge = prewrite both adjacency sides with primary on u, supernodes are the Zipf head — per-shard adjacency segments turn the WW hotspot into scatter-gather reads, every protocol step gets a kill point with no-dangling-half-edges as the invariant.
• 2026-07-10 — topic 28 scaffolded: study guide (the latency ladder measured and priced — local NVMe p50 0.10 ms vs raw S3 p50 14.17 / p99 112.99 ms = 140× median, 940× tail, and why everyone moved anyway: $/GB, 11-nines durability = replication-is-someone-else’s-problem, scale-to-zero; the 2008→Snowflake→Aurora→Socrates→Neon→SlateDB lineage mermaid; Neon’s four-box data flow with the safekeepers-commit-fast / pageserver-serves-pages split; five-system design-space table with what-crosses-the-network as the axis; WAL-rule-promoted-to-architecture rosetta linking topic 27’s Kafka thesis), 5 reading guides (Aurora SIGMOD ’17 — the-log-is-the-database, 6-copy 4/6 AZ+1 quorums over 10 GB protection groups, 35× network amplification killed, VDL-replaces-2PC question, commit=log-quorum-ack + recovery-without-REDO-at-compute; Socrates SIGMOD ’19 — durability≠availability as THE decomposition, XLOG landing zone vs page servers vs XStore mapped onto topic 5’s WAL lifecycle and onto Neon components, RBPEX = buffer pool made restart-durable; Snowflake SIGMOD ’16 + Building-a-DB-on-S3 SIGMOD ’08 paired — the prescient paper’s three blockers (eventual consistency/no CAS/request cost) vs what fixed each (strong consistency 2020, conditional PUT 2024, immutability routed around all three), micro-partitions as CoW-clone-by-file-list, min/max pruning = topic 26’s BRIN at cloud scale; Neon code walk — get_rel_page_at_lsn pgdatadir_mapping.rs:258 → Timeline::get :1227 → LayerMap::search layer_map.rs:448 as an LSM over (page, LSN), walredo.rs:173 = REDO on the read path in a sandboxed Postgres, branch_timeline_impl tenant.rs:4985 O(1) branches + the timeline.rs:4548 ancestor walk our stub reimplements, branch-aware GC retain-point question; SlateDB+Quickwit S3-first code walk — tablestore.rs:835 block-granular ranged GETs, cached_object_store part cache = our cache.rs in production form, fence.rs:105 CAS-epoch fencing = consensus outsourced to S3 conditional PUT, clone.rs:38 zero-copy clones, quickwit bundle hotcache footer + TimeoutAndRetryStorage :37 hedging with the AWS-recommends-it citation, pathology→countermeasure convergence table), experiments crate compiles: virtual-latency sim (charged-not-slept lognormal S3 with 2% 8× stragglers, NVMe, scripted Fixed model for exact contract tests) + block store + zipf + percentiles PROVIDED — 6 provided tests pass, tier_bench provided lanes RUN (numbers above) — LruBlockCache+TieredReader (touch-protects, 1/8-cache zipf hit-rate >50%, block-sharing hits), hedged_get (scripted 50ms-primary/1ms-backup/10ms-deadline ⇒ exactly 11ms, p99-halves-at-<10%-extra-GETs on straggler-heavy S3), BranchStore::get (parent-prefix visibility, sibling isolation, PITR historical branch points, 100-deep chains, branching-copies-nothing proven by version_count) are todo!() stubs — 13 tests fail as todo panics; notes.md predicts cache-fixes-the-median-hedging-fixes-the-tail and flags the O(n)-eviction-scan wall-time trap; M28 log: L0+WAL stay local (landing-zone lesson), manifest-CAS fencing not leases, branch at SST-list not page granularity, image-layer materialization deferred until ancestor walks profile hot.
• 2026-07-10 — topic 27 scaffolded: study guide (recompute-is-the-enemy with the priced motivation table — full recompute per 100-change batch: triangles 97.2 ms / wedge self-join 894.3 ms / re-BFS 24.7 ms vs µs-scale incremental targets; the one algebraic idea as a table — LINEAR ops stream deltas statelessly, BILINEAR joins need arranged inputs via Δ(A⋈B)=ΔA⋈B+A⋈ΔB+ΔA⋈ΔB, NONLINEAR distinct/aggregates need integrals; DBSP I→Q→D mermaid; timestamps/watermarks section: timely frontiers are proofs where Flink watermarks are heuristics; four-system comparison timely/DBSP/Materialize/RisingWave), 5 reading guides (Naiad+timely — could-result-in pointstamp protocol, MutableAntichain frontier.rs:380/update_iter :533, ChangeBatch :16 progress-updates-are-Z-set-shaped, worker.rs:235 step as the topic-7 event loop one layer up, the rosetta table frontier=vacuum-watermark; differential — consolidation.rs:24 = our from_updates verbatim, arrangements as LSM-of-batches with advance=compaction, join_traces join.rs:69 with the fuel/effort loop :348-395 as operator-level cooperative yielding, iterate.rs:192 Variable + bfs.rs:101-107 as the 40 lines our stub can’t do, why deletion-in-recursion needs lattice times; DBSP — Q^Δ=D∘Q∘I with the chain rule as the compositional bombshell, feldera anchors z1.rs:221/integrate.rs:85/differentiate.rs:38/join.rs:123-350/delta0.rs, nested-circuits-vs-lattice trade, the M27 mapping: delta matrix DP−DM=ΔA and wedges need NO new state because the integrals ARE the adjacency matrices; Materialize+RisingWave — dogs^3 delta_join.rs:47 + half_join :315/:402 avoiding intermediate arrangements, indexes-are-arrangements-are-memory, RisingWave Op enum stream_chunk.rs:45 as Z-set-weights-as-protocol, hash_join.rs:158 degree tables :269 as hand-rolled weight bookkeeping vs one consolidation rule, barrier checkpoints, single-writer-gets-the-hard-parts-free table; Kafka NetDB’11 — dumb-broker/smart-consumer, offset=LSN rosetta, log-compaction=arrangement-advance=LSM-GC same operation three communities, exactly-once = where-do-offsets-live, M27’s raw-log-vs-result-delta subscriber decision), experiments crate compiles: ZSet with consolidation + the distinct-is-not-linear load-bearing test, churn generator with set-semantics guard, full-recompute oracles (sorted-intersect triangles, hash join with weight multiplication, BFS) PROVIDED — 6 provided tests pass, ivm_bench baselines RUN (numbers above) — delta_join+IncrementalJoin (algebra-exact vs join(A+ΔA,B+ΔB)−join(A,B), 30-batch drift-free, deletes-retract), IncrementalTriangles (oracle-tracking under churn, K4-minus-edge=−2, <4K probes for 20 changes on 40K edges), SemiNaiveReach (insert-only BY DESIGN — deletion is differential’s lattice territory, documented; matches re-BFS per batch, ≤4 relaxations/edge EVER, intra-component edges free) are todo!() stubs — 9 tests fail as todo panics; notes.md flags the honest suspicion that IncrementalJoin’s Vec-merge state integration may dominate and re-derive why arrangements exist.
• 2026-07-10 — topic 26 scaffolded: study guide (indexes-are-bets framing with the measured motivation table — point-miss binary search 167 ns / BTreeMap 218 / HashSet 24 at 224 MB, vs blocked bloom’s ~15-25 ns at 12 MB target; three-families ASCII filters/sketches/learned; bloom math block with the reproduce-it FPR derivation; bloom→blocked→cuckoo→xor→ribbon lineage mermaid; HLL sparse-30-bytes-to-dense-12KB story; PGM ε-window thesis), 6 reading guides (bloom→ribbon over RocksDB code — FastLocalBloomImpl bloom_impl.h:144 golden-ratio probe remix vs LegacyBloomImpl :364, CacheLocalFpRate :42 as the Poisson-crowding honesty function, ribbon as banded GF(2) solve with StandardBanding ribbon_impl.h:471 / num_starts_=slots−kCoeffBits+1 :504 and the build-can-fail-vs-monotone question; cuckoo+xor — partial-key involution getAltHash cuckoo.c:122, RedisBloom’s LSM-of-subfilters growth CuckooFilter_InsertFP :256 vs the paper’s fail-at-MAX_KICKS, delete-a-false-positive-corrupts-someone-else contract, xor peeling 1.23× vs bloom 1.44× vs ribbon 1.10×; HLL — hllPatLen :467 / Ertl tau+sigma :1016/:1033 replacing HLL++’s empirical bias tables, ZERO/XZERO/VAL sparse opcodes :380 with promote-at-3KB-or-rank>32, merge-is-a-semilattice AVX2 :1116; learned indexes — Kraska RMI’s no-error-bound flaw, PGM_SUB/ADD_EPS pgm_index.hpp:32-33 + optimal hull PLA piecewise_linear_model.hpp:96/:154-190 vs our simpler shrinking cone, ALEX gapped arrays predict_position alex_nodes.h:1448 + exponential search :1462 with the degrades-in-space-vs-time-vs-write-amp scoreboard; roaring internals extending topic 23 — Store enum store/mod.rs:28-31, ARRAY_LIMIT=4096/RUN_MAX_SIZE=2048 as pure arithmetic container.rs:9-11, 3×3 pairwise kernel dispatch, three-adaptive-encodings table roaring/HLL-sparse/GIN-varbyte; postgres indexam as the classical baseline — _bt_search nbtsearch.c:100 + Lehman-Yao moveright :211, GIN varbyte ginCompressPostingList ginpostinglist.c:196 + pending-list-as-mini-LSM, BRIN bringetbitmap brin.c:301 as the one-sided filter that’s 10,000× smaller than bloom when clustering holds), experiments crate compiles: splitmix64/hash2/fastrange PROVIDED (avalanche + coverage tests pass), filter_bench motivation lanes RUN — binary search miss 167 ns ≈ 23 dependent misses, BTreeMap 218, HashSet 24 ns at 224 MB, hit≈miss 169 (the walk is the cost, not the compare) — BlockedBloom (no-FN + FPR<2.5%@10bpk + <4×-theory + halves-8→16 tests), CuckooFilter (no-FN at 90% load, FPR<1%@12-bit, delete-leaves-others-intact — the test bloom can never pass, graceful-full-failure), Hll (err<3% at 1K/100K/5M, merge registers EXACTLY equal union’s), and LearnedIndex (window ≤2ε+2 always contains true pos, uniform-1M <2K segments, ε holds on hostile powers-of-2+quadratic mix) are todo!() stubs — 15 contract tests fail as todo panics, notes.md has predict-before-measure table and the M26 call that learned indexes are NOT in scope (node IDs are dense — a plain array is already the perfect model).
• 2026-07-10 — topic 25 scaffolded: study guide (message-passing-is-SpMM with receipts — the message_and_aggregate table showing GCNConv = spmm(adj_t, x) at gcn_conv.py:273 and SAGEConv = spmm(..., reduce=mean) while GAT can’t fuse because attention recomputes the matrix values per forward; associativity-as-query-plan — (AX)W vs A(XW) swaps which term carries the big dimension, 90× on Cora; GraphRAG closed-loop mermaid graph→embeddings→M14 vector index→hybrid Cypher), 7 reading guides (node2vec KDD ’16 — second-order walk figure, p/q as BFS↔DFS knobs, the alias-table O(m·avg_deg) memory trap with rejection sampling as the fix, PyG Node2Vec.loss node2vec.py:135 as the SGNS reference; Kipf-Welling GCN — renormalization trick, gcn_norm anchors gcn_conv.py:45-71, GCN-forward-is-a-query thesis, oversmoothing as why-2-layers; GraphSAGE — sampling as a page budget (B·10·25 fan-out math), mean+lin_r≈concat, inductive-is-the-only-write-friendly-variant; GAT — SDDMM+softmax+SpMM kernel decomposition = topic 24’s masked SpGEMM, a_src/a_dst per-node split as factor-out-of-join, materialized-vs-computed view line running exactly between GCN and GAT; PyG message-passing machinery — 10-stop code walk, COO gather-scatter materializes an m×d temp (145 MB on our bench) vs CSR spmm’s zero temporaries = materialize-the-join vs pipeline-the-aggregate, message()-as-arbitrary-callable = Ligra’s F-with-CAS tradeoff third community; TransE — relations as translations, symmetric-relation collapse, link-prediction-is-an-ANN-query so M14 serves KG completion natively; GraphRAG-SDK with systems eyes — vector_store.py:344 queryNodes / :219 SET embedding write path, relationship_expansion’s ANN+k-MATCHes as the k+1-round-trip join to push down, multi_path’s client-side cosine rerank, router-as-planner-with-no-cost-model, four systems smells), experiments crate compiles: CSR + SBM generator (ground-truth labels; O(m) inter-block sampling not O(n²) Bernoulli) + ring-of-cliques + dense Mat/glorot/softmax + SpMM + row-norm adjacency + uniform walks + dense GCN oracle PROVIDED and run — SBM 16,384 vertices/566K edges: uniform walks 42.8 Msteps/s, SpMM 21.2 GFLOP/s = 81% of dense matmul’s 26.2 (64-wide feature rows amortize the gather — fat RHS forgives sparsity, the number that makes M25 plausible) — node2vec_walks (4 tests: degree-stationary dist, p=q=1 ≡ uniform, q orders exploration on ring-of-cliques, p orders backtrack rate), train_skipgram (SBM intra-block cosine must beat inter by 0.2), gcn_norm+gcn_forward (dense-oracle 1e-4, sorted rows, transform-before-aggregate) are todo!() stubs.
• 2026-07-10 — topic 24 scaffolded: study guide (per-source vs whole-graph algorithm map; frontier-world vs algebraic-world mermaid with the honest trade — per-vertex tricks like Afforest’s edge skipping vs LAGraph’s batched matrix frontiers and atomics-free bulk ops; rmat-vs-uniform baseline table making skew the headline), 6 reading guides (GAP arXiv:1508.03619 + gapbs anchors — sssp.cc:44’s redundant-relaxation-beats-bookkeeping bet, bc.cc:76 succ bitmap, cc.cc:69/:106/:129 Afforest’s three phases, tc.cc WorthRelabelling, and the 5-graph matrix as topic 22’s change-anything-different-number; Meyer-Sanders delta-stepping as the Dijkstra↔Bellman-Ford dial with gapbs thread-local-bins vs LAGr_SSSP’s MIN_PLUS tmasked vxm + three implementation traps; Brandes ’01 with the dependency-recurrence derivation exercise + gapbs-vs-LAGr_Betweenness table — the batched ns×n matrix frontier amortizes what frontier code cannot; Ligra PPoPP ’13 — edgeMapData ligra.h:235-272, the m/20 threshold :238, dense/sparse/denseForward, PageRank-degenerates-to-SpMV lesson, m/20 = Beamer α/β = dot-vs-saxpy; Louvain→Leiden Sci Rep ’19 — the disconnected-communities bug as topic 21’s greedy-destructive trap, refinement as egg’s keep-both-forms, ΔQ accumulator = SPA, aggregation = S·A·Sᵀ SpGEMM, determinism-needs-seeding for CALL algo.community; LAGraph analytics — FastSV7 mngp/hooking-as-one-mxv :102 + FASTSV_SAMPLES :335, TriangleCount’s six formulations with the :44 urand-flips-to-saxpy exception, PageRankGAP-vs-PageRank as benchmark-specs-fork-implementations, and FalkorDB’s proc_pagerank.c:197 already calling LAGr_PageRank = M24’s pattern exists, re-plumb it), experiments crate compiles: weighted CSR + RMAT/uniform generators + heap Dijkstra with pop counter + pull PageRank + degree-ordered triangle count + union-find CC + O(n³) definitional BC oracle PROVIDED and run — RMAT scale 16 vs uniform same n/m: 15,645,988 vs 5,428 triangles (2,883×) in 376/158 ms, PR 8-vs-6 iters (hubs slow L1 decay), Dijkstra 343K pops = 1.74×n stale-entry tax, 18,844 components at avg_deg 16 (RMAT’s leaf quadrant strands vertices) — delta_stepping (bucketed, extremes-must-still-be-exact test + relaxation counters), brandes (must match the O(n³) oracle exactly on n=128, then sample GAP-style), and afforest (partition-equal + <50%-of-m edges-inspected bound) are todo!() stubs; RMAT skew assertion needed scale-aware bounds (19.1% top-1% share at scale 12, 36.6% at 16).
• 2026-07-10 — topic 23 scaffolded: study guide (inverted-index anatomy ASCII — analyzer → FST term dict → TermInfo{doc_freq, postings_range} → 128-doc Δ-bitpacked blocks with {last_doc, max_score} skip data; write-path mermaid making Lucene-segments-are-an-LSM explicit — tiered LogMergePolicy works for text because queries fan out anyway; the two speed tricks: compression-with-random-access + score upper bounds), 6 reading guides + 2 cross-linked (Zobel-Moffat CSUR ’06 design-space map — TAAT vs DAAT, capped accumulators as 2006’s WAND, merge-based construction = LSM before Lucene; Robertson-Zaragoza BM25 derivation ladder — eliteness ⇒ tf saturation at K1+1 ⇒ the static ceiling WAND needs, mapped to tantivy bm25.rs:8-59 with the 1-byte-fieldnorm quantization question; Ding-Suel SIGIR ’11 block-max WAND with pivot diagram + four implementation traps (θ seeding, livelock-on-failed-refinement, k-boundary ties, docs-evaluated metric); Roaring — 4096 crossover derivation, kernel matrix, containers = GraphBLAS sparse↔bitmap lattice at 64K granularity; tantivy code walk — compression/mod.rs:3 128-blocks, skip.rs:93/:175/:186 SkipReader/block_max_score, term_info.rs:9-13, fst_termdict, block_wand_union.rs:8-24 find_pivot_doc, log_merge_policy.rs:20-24, 90-minute read order; RediSearch redisearch_rs Rust-rewrite — newly cloned — InvertedIndex core.rs:30/:75 chained varint IndexBlocks vs tantivy’s immutable bitpacked segments, Encoder-as-type-parameter monomorphizing 11 codecs, gc_marker/unique_id cursor validation ↔ delta-matrix wait, new-block-on-delta-overflow, and the finding that RediSearch has NO block-max WAND — scored unions walk everything), experiments crate compiles: zipf corpus (term id = rank, df(t0)=99.9%) + tf-counting index builder with per-128-block max-BM25 metadata + saturating-BM25 + exhaustive TAAT oracle PROVIDED and run — 100K docs/7.9M postings built in 335 ms, common∧rare [t0 t12000] top-10 walks 99,964 postings in 6.34 ms at ~32 ns/posting (hash accumulate dominates — Q1’s lesson again) while the rare term carries ~93% of the winning score = the WAND poster child measured, and vec two-pointer dense∧sparse AND costs O(|dense|) 52 µs for 172 hits = the roaring motivation measured — block-max wand_topk (recipe in doc comment, must match oracle top-k while scoring <25% of the postings) and mini-Roaring (array/bitmap containers, 3 density-crossing oracle tests) are todo!() stubs.
• 2026-07-10 — topic 22 scaffolded: study guide (OLTP↔OLAP benchmark map + the number-is-four-choices mermaid workload/data/harness/metric; choke points one-liner table — Q1 tiny-group agg = expression bench, Q6 2%-scan = the GB/s headline, Q9 = optimizer punisher, TPC-C = the D_NEXT_O_ID hot counter institutionalized; benchmarking-sins checklist linking topic 0’s fair-benchmarking guide), 4 new reading guides + 3 cross-linked existing ones (Boncz TPCTC ’13 choke-point taxonomy with duckdb dbgen/queries dir open + hidden messages — uniform data is why JOB exists, Q1’s 4-6 groups make hash-agg invisible; YCSB SoCC ’10 + go-ycsb zipfian.go:92-165 anchors — zetan/eta/alpha math, the two fast paths, scrambled-fnv rationale, coordinated-omission warning; OLTP-Bench VLDB ’13 + benchbase TPCC anchors — keying/think times :85-100, NURand C_LAST load-vs-run constants :94-116, why nobody runs TPC-C honestly = 12.86 tpmC/warehouse; DuckDB tpch extension — dbgen as streaming TABLE FUNCTION tpch_extension.cpp:17-99 with answers/ shipped next to queries/ = benchmark-as-oracle, run-real-TPC-H-here recipe), experiments crate compiles: dbgen-lite lineitem + row-at-a-time Q1/Q6 oracles + YCSB A-F driver over BTreeMap with ns-percentile Hist PROVIDED and run — Q1 oracle 5.6 GB/s effective (HashMap per row even at 6 groups: CP1.2 measured), Q6 branchy oracle 15.7 GB/s (2% selectivity = perfectly-predicted branch, the crater is hiding at 50%), YCSB uniform A-F 2.88/4.15/3.72/4.40/1.11/2.85 Mops/s with E’s scans 4× a point read — Zipfian/Scrambled generators (the actual YCSB math with head-frequency-vs-theory statistical contract tests) and q1_flat/q6_branchless columnar lanes are todo!() stubs; fixed a nearest-rank percentile off-by-one in the harness itself (harness bugs are results bugs).
• 2026-07-10 — topic 21 scaffolded: study guide (tool-per-guarantee table proptest→TLC→SMT→Lean with cost axis; e-graph = union-find + hashcons + congruence closure with egg’s deferred rebuild = delta-matrix wait = LSM compaction — batch the invariant repair; equality-saturation loop mermaid; TLA+ spec-as-math with the measured state counts; Z3 DPLL(T) diagram + the euf_egraph.h:23 comment where Z3 cites egg back; Perceus RC as the Arc::make_mut compiler pass), 5 reading guides (AWS CACM ’15 — 35-step S3 bug, exhaustively-testable-pseudo-code pitch, small-scope hypothesis; egg POPL ’21 with full source anchors — egraph.rs:970 add / :1147 union / :1416 rebuild / :1346 process_unions fixpoint, machine.rs Bind/Scan/Compare pattern VM = topic 19’s bytecode interpreter, extract.rs greedy find_best vs lp_extract, e-graph ≈ Cascades memo; Z3 TACAS ’08 + src/ast/euf anchors — backtracking trail + justifications = WAL for unions, e-matching triggers = index choice; Specifying Systems + Ongaro’s raft.tla — newly cloned — with the un-model-an-assumption exercise that re-derives terms; Beans + Perceus borrowed-vs-owned + reuse tokens with the proof-vs-TLC-vs-proptest calibration exercise), experiments crate compiles on egg 0.9: expr IR + hand-ordered fixpoint rewriter PROVIDED and run — ~30% cost reduction, ~2 µs/firing, and the planted trap measured: (a*2)/2 → strength-reduce fires before div-reassoc → stuck at (a<<1)/2 cost 5 where egg should reach cost 1 — egg_optimize is a todo!() stub with trap + never-worse-than-hand tests; TLA+ WalReplication spec written AND model-checked (tla2tools.jar downloaded, java 17): SyncCommit=TRUE → Durability holds over 1080 distinct states depth 14 in <1 s; SyncCommit=FALSE → TLC finds the 5-state data-loss trace Append→Commit→Crash→Failover after 123 states — the postgres synchronous_commit=off story, found exhaustively.
• 2026-07-10 — topic 20 scaffolded: study guide (format lattice hypersparse→sparse→bitmap→full with the actual switch tests from GB_convert_sparse_to_bitmap_test.c and GB_conform; one-mxm-four-engines mermaid — dot3 iterates the MASK so work ∝ nnz(M) vs saxpy3’s coarse/fine × Gustavson/hash task scheduler with its flopcount pre-pass = cudf size/retrieve five years early; push-vs-pull BFS as vxm-vs-mxv with LAGraph’s shipped α=8/β1=8/β2=512; delta matrices as LSM-over-matrices — DP=memtable, DM=tombstones, wait=minor compaction, delta_mxm’s (A*(M+DP))<!A*DM> fold), 6 reading guides (Davis TOMS ’19+’23 — zombies/pending as the library’s own deltas, iso matrices, 32-bit indices; SuiteSparse internals with saxpy3.c:22-60 scheduling-essay and hash>m/16⇒Gustavson anchors; Gustavson ’78 + Buluç-Gilbert — SPA design space, symbolic/numeric two-phase; Beamer SC ’12 direction-optimizing with the ICPP ’18 linear-algebra translation; LAGraph — BFS template switch block anchors, ANY_SECONDI parent-without-comparisons, six triangle-count formulations, PageRankGAP; FalkorDB delta_matrix with fresh eyes — header state-table as spec, transposed twin, transpose-as-masked-copy sync, over-masking question — LAGraph newly cloned), experiments crate compiles: CSR+RMAT/uniform/path generators, SpMV, hash-SpGEMM, scalar BFS, hypersparse PROVIDED and run — SpMV 16-19 GB/s single-thread (gather tax vs 30 GB/s streaming), hash SpGEMM ~60-75 Mflop/s (15 ns/flop = the accumulator cost the SPA stub should crush), hypersparse 50× index memory and 171× full-sweep on the 10M-nodes/100K-edges FalkorDB shape — dense-SPA Gustavson and push/pull/direction-optimizing BFS (with per-level trace + path-graph-never-pulls test) are todo!() stubs.
• 2026-07-10 — topic 19 scaffolded: study guide (the spectrum tree-walker → bytecode VM → copy-and-patch → IR JIT → LLVM as a compile-latency-vs-run-speed trade with each system placed on it; produce/consume compiles the PIPELINE not the operators — push inverts control so tuples stay in registers; three JIT grains compared — postgres per-query, Umbra per-pipeline with adaptive Flying-Start→LLVM tiering, GraphBLAS per-kernel-specialization cached forever; DuckDB’s deliberate no-JIT with the VLDB ’18 tie as the counter-argument; M19 = expressions only, gate on measured cost), 6 reading guides (Neumann VLDB ’11 pipelines/breakers + produce-consume inversion; SQLite VDBE — vdbe.c:1049 switch over 199 opcodes, register machine, OP_Yield coroutines as flattened-bytecode’s free resumability; Umbra Tidy Tuples single-pass IR + copy-and-patch musttail stencils; postgres llvmjit_expr.c opblocks-per-EEOP-step + the jit_above_cost estimate-gate failure taxonomy + deform-JIT-as-the-real-win; GraphBLAS jitifyer encodify-hash → PreJIT → memory table → dlopen → invoke-cc ladder with FalkorDB cache-warming implications; cranelift-jit-demo declare→define→finalize→transmute ladder + per-node CLIF emission table for the stub), experiments crate compiles on cranelift 0.116: Expr enum + seeded generator, AST interpreter, and column-at-a-time vectorized lane PROVIDED and run — interp ~2.1 ns/node flat, vectorized 6-12× over interp across depths 2-10 (topic 11’s number reproduced; both linear in nodes, no y-intercept until compile time adds one) — jit.rs compile() is a todo!() stub with bit-exact-vs-interpreter tests, jit_bench prints the three-way table with compile µs + e2e winner and survives the stub via catch_unwind.
• 2026-07-10 — topic 18 scaffolded: study guide (GPU-for-DB-people translation table — SIMT = topic 17’s predication in hardware, coalescing = columnar × 32, shared memory = cache blocking made explicit; the bus decides the architecture — Crystal’s regime A ship-per-query vs regime B device-resident, rewritten for Apple unified memory; libcudf size/retrieve two-phase + cooperative-groups probing = SwissTable at warp scale; Gunrock advance/filter + load-balance menu; CAGRA = HNSW with SIMT-hostile parts deleted by construction), 6 reading guides (Crystal SIGMOD ’20 tile model + fair-CPU-baseline lesson; wgpu compute examples ladder incl. the hello_compute doc-comment our bench proves; libcudf join size/retrieve + shared-mem-until-spill groupby with cuco/cooperative-groups anchors — newly cloned; Gunrock Essentials bfs.hxx enactor + advance.hxx thread/block/merge_path dispatch anchors — newly cloned; CAGRA ICDE ’24 + single-CTA kernel/shared-mem-hashmap/bitonic-topk anchors — cuvs newly cloned; Faiss GPU billion-scale paper — WarpSelect register k-select, memory-tier table), experiments crate compiles AND RUNS on Metal: GpuCtx + workgroup-reduction sum kernel PROVIDED with per-phase timings, gpu_bench crossover sweep run on Apple M3 Pro — CPU wins at every size up to 16M elements (~1.5 ms fixed dispatch floor, flat 16K→1M; even amortized the GPU reads the same unified memory at ~9 GB/s effective vs CPU 30 GB/s — regime B’s bandwidth ratio doesn’t exist for streaming ops on this machine, which IS the lesson), filter_count (one-atomicAdd-per-workgroup, WGSL skeleton provided) and l2_batch (one-invocation-per-target + row/column-major coalescing experiment) are todo!() stubs with exact-match/1e-3 tests, notes.md predicts where arithmetic intensity finally flips the verdict.
• 2026-07-10 — topic 17 scaffolded: study guide (ports×latency mental model — M-series 4 FMA ports × 3cy ⇒ ~12 independent chains, the four autovectorization failures, branchy/branchless/compress filter shapes with AVX-512 vpcompress vs NEON’s missing compress, vshrn movemask idiom, FastLanes interleaved layout), 7 reading guides (simdjson VLDB ’19 + arm64 nibble-LUT classification / PMULL prefix_xor quote parity / LUT-shuffle compress emulation / flatten_bits over-write-under-advance — newly cloned; polars-compute float_sum STRIPE=16 + pairwise-128 and simd_filter! with per-ISA compress + selectivity-adaptive scalar bit-iteration fallback; hashbrown Group×3 backends + memchr Vector — newly cloned — with the finding that hashbrown’s NEON group is 8 BYTES so vceq output already IS the bitmask, no vshrn, while memchr keeps 16 lanes and narrows; SimSIMD under its numkong rename — per-instruction latency/port tables in headers, f64-upcast accumulation, 4-target streaming states = M14’s scoring loop, and the FCMLA lesson: the specialized instruction measured 2.3× SLOWER than 4 plain FMAs; SIGMOD ’15 selective-store/load primitives + vertical hash probing + gather-costs-a-load-per-lane; FastLanes VLDB ’23 1024-lane transposed layout; Mojo SIMD[type,width] parametric-width ladder), experiments crate compiles: dot naive+unrolled-8 PROVIDED and run — 10.89 → 42.12 GB/s, 3.9× from accumulator count alone, zero intrinsics — wide-f32x4 and NEON vfmaq 4-accumulator rungs are todo!() stubs; filter branchy+branchless PROVIDED and swept — branchy craters 9× to 1.19 GB/s at 50% selectivity while branchless holds ~12.7 flat, the SIGMOD ’15 curve live — NEON count (vcltq+vsubq mask-accumulate) and LUT-compress compact (simdjson trick, f32 edition, all-16-masks test) are stubs; unpack4 scalar PROVIDED at 10.20 GB/s, NEON shift/mask stub; simd_bench catches stub panics so baselines always print.
• 2026-07-10 — topic 16 scaffolded: study guide (every technique = generator + oracle table, DST determinism boundary diagram, PQS/TLP/NoREC comparison, Jepsen/elle, Z3-as-search-engine), 6 reading guides (turso testing/simulator with clock/io/file fault-injection anchors + interaction-plan properties + doublecheck + structured fuzz targets; FDB simulation + BUGGIFY + Antithesis determinism-boundary table; SQLancer oracle base classes — newly cloned — PQS check()/rectification, TLP 3-way partition, NoREC optimized-vs-forced-scan; PQS OSDI ’20 + TLP OOPSLA ’20 paired; Jepsen redis-raft/Dgraph findings + elle cycle inference; Z3 — newly cloned — TACAS ’08 + tactic/solver/smt_context anchors + Cosette symbolic-row rewrite verification), experiments crate compiles: sim_fs (buffered/synced/torn-tail file) + kv (WAL KV with 4 injectable bugs: LostDelete/NoSyncOnCommit/TornWriteAccepted/StaleRead) PROVIDED, dst harness + ddmin shrinker + TLP Kleene-eval checker are todo!() stubs with 12 contract tests (all bugs caught ≤200 seeds, zero false positives over 500, deterministic replay, 1-minimal repro, null-blind engine exposed), crash_matrix PROVIDED and run: 5000 seeds/0.02 s per bug — 0.0% false positives, bugs caught at 48.8–99.6% per-seed rates, first failing seed ≤ 3; the matrix caught a real bug in this crate’s own recovery (missing WAL tail truncation made torn leftovers join the next batch — 72.7% divergence until fixed), the topic’s thesis self-demonstrated.
• 2026-07-10 — topic 15 scaffolded: study guide (topology menu as WHO-can-ack axis, Raft state-machine mermaid + election/log-matching/§5.4.2 three-way split, write-path comparison valkey/WAIT/raft, consistency ladder + ReadIndex, hash slots vs ranges), 6 reading guides (Raft ATC ’14 with Fig 8 worked by hand; valkey replication.c shared repl buffer + PSYNC replid/offset + REPL_STATE_ handshake + WAIT + FAILOVER with line anchors; tikv raft-rs — newly cloned — RawNode/Ready contract + step_* dispatch + Progress tracking + maybe_commit-is-§5.4.2; qdrant consensus.rs raft-for-metadata-only split with the data path outside raft; VSR Revisited round-robin views + no-disk durability + TigerBeetle disk-can-lie; DDIA ch. 5/8/9 anomaly catalog + fencing tokens + linearizability), experiments crate compiles: sim.rs deterministic lockstep network PROVIDED (seeded delivery, partition/heal — topic 16 DST preview), raft.rs todo!() stub with 5 safety-pinning tests (one leader, one-per-term across 10 seeds, replicate-to-all, minority-commit-freeze, stale-leader truncation), partition_test timeline binary, repl_lag PROVIDED and run: follower fsync policy AS ack latency — every-1 = 339 entries/s at 2973 µs p50 (F_FULLFSYNC) vs never = 18568/s at 6 µs, the topic 5 ladder measured as replication lag.
• 2026-07-10 — topic 14 scaffolded: study guide (recall-vs-QPS curve framing, HNSW-as-skip-list ASCII, quantization ladder u8/PQ/binary with oversample+rescore, IVF + DiskANN families, filtered-search menu with percolation), 6 reading guides (qdrant GraphLayers builder/serve split + visited pool + the per-query algorithm choice HNSW/ACORN/plain via estimate_cardinality + measured percolation at build; qdrant quantization crate u8 affine dot-expansion / PQ ADC LUTs / binary xor_popcnt + get_oversampled_top; usearch — newly cloned — node-tape layout + paper-default constants + striped locks (helix-db dropped: public repo no longer ships engine source); HNSW paper with skip-list lens; Jégou PQ with SDC/ADC + IVFADC residuals-as-FOR; DiskANN Vamana robust-prune α-slack + PQ-steers/f32-ranks SSD layout), experiments crate compiles: brute-force oracle PROVIDED and run (185 QPS at recall 1.0 over 100K×128-d — the floor), hnsw (Alg 1/2/4, level draw, ef knob) and quant (affine u8 + symmetric distance + rescore pipeline) are todo!() stubs with 10 contract tests (self-query top-1, recall@10 ≥ 0.9, sorted results, log level distribution, α/2 error bound, rescored recall ≥ 0.95), ann_bench sweeps ef 16..256 + oversampling 1/2/4.
• 2026-07-10 — topic 13 scaffolded: study guide (adjacency representation menu with CSR ASCII, four-architecture table neo4j/memgraph/kuzu/FalkorDB across store/Expand/pattern-match/MVCC/updates, delta-overlay-as-LSM observation, pointer-chasing cost analysis, WCOJ/AGM section, LDBC referee), 6 reading guides (GraphBLAS 4 sparsity formats + dot-vs-saxpy mxm + masks-as-pushdown + FalkorDB Delta_Matrix M/DP/DM state machine — neo4j/kuzu/GraphBLAS newly cloned; neo4j 15 B node / 34 B rel fixed records + doubly-linked rel chains = one miss per edge; memgraph skip-list vertex + small_vector edges + PointerPack’d delta MVCC; kuzu columnar CSR node groups persistent+transient + Intersect WCOJ + factorization; AGM bound / Generic Join / EmptyHeaded with the C<A>=A² equivalence; LDBC SNB correlated-power-law datagen + updates-during-reads), experiments crate compiles: adj_list oracle PROVIDED and run (1M-node/16M-edge preferential-attachment: 3.5 µs/query random vs 295 µs supernodes — the 85× graph-shaped tail, max degree 6565), csr (counting-sort build + slice two_hop) and matrix (masked-SpMV two_hop) are todo!() stubs with oracle-agreement + exact-layout + cycle-self-exclusion tests, hop_bench cross-checks via checksums.
• 2026-07-10 — topic 12 scaffolded: study guide (row-vs-column ASCII, lightweight-encoding zoo table incl. FSST, analyze→score→compress lifecycle, zone-map pruning diagram, Arrow-vs-Parquet boundary, MergeTree/DuckDB/Pinot architecture table), 6 reading guides (DuckDB compression framework + 4-mode bitpacking + fetch_row-shapes-the-menu + CheckZonemap; ClickHouse MergeTree — newly cloned — parts/granules/sparse-index/marks two-offset trick + merge-time work; arrow-rs + parquet-rs — newly cloned — buffer recipes, RLE-hybrid, two compression layers; C-Store + SIGMOD ’06 process-compressed thesis; BtrBlocks sampling cascade + FSST symbol tables; ClickHouse VLDB ’24 with ClickBench-on-DuckDB exercise), experiments crate compiles: RLE/Dict/BitPacked todo!() stubs with exact-size + maximal-runs + FOR-width + O(1) random-access contract tests, scan_bench PROVIDED (100M values × 3 shapes, raw vs encoded scans incl. RLE sum-without-decode and dict codes-only sum — “raw-equiv GB/s > memory bandwidth” is the compression-IS-performance headline to verify).
• 2026-07-10 — topic 11 scaffolded: study guide (Volcano→X100→HyPer mermaid, selection vectors + vector-type flags, morsel-driven parallelism diagram, vectorized hash join/agg internals), 6 reading guides (DuckDB DataChunk/2048 + pipeline executor push-pull hybrid + join-HT salt-in-pointer probe; postgres ExecProcNode self-replacing dispatch + execExprInterp computed-goto; polars-stream Morsel/MorselSeq/SourceToken + float_sum masked-SIMD multi-accumulator + DataFusion ExecutionPlan streams and intern-then-flat-arrays GroupedHashAggregateStream; X100 CIDR’05 U-curve; VLDB’18 compiled-vs-vectorized scorecard — memory-bound probes favor vectorized, the M11 architecture argument; SIGMOD’14 morsels), experiments crate compiles: one query three engines — Volcano PROVIDED and run (180.7 M rows/s; found LLVM DEVIRTUALIZING the statically-known Box<dyn> chain, 202→180 after black_box — a compiler will silently turn your Volcano into a compiled engine), vectorized (batches + selection vectors + flat group array) and fused branchless kernel are todo!() stubs with oracle-agreement tests incl. partial-final-batch and mask-sign-extension traps, exec_bench sweeps selectivity 5/50/95.
• 2026-07-10 — topic 10 scaffolded: study guide (parse→bind→logical→rewrite→join-order→physical pipeline mermaid, rewrite-rule menu, Selinger DP vs DuckDB DPccp+greedy-fallback, cardinality three-lies table, Selinger-vs-Cascades memo ASCII), 5 reading guides (DuckDB optimizer.cpp 25-pass pipeline + plan_enumerator DPccp with greedy escape hatch :234 + cost=output-cardinality-only; postgres allpaths.c standard_join_search + geqo threshold 12 + DEFAULT_EQ_SEL 0.005; sqlparser-rs Pratt parse_subexpr + DataFusion fixpoint-of-rules vs DuckDB ordered passes — sqlparser/datafusion/polars newly cloned; Selinger ’79 vs Cascades with M10 architecture-choice question; Leis VLDB’15 JOB — cardinality error 10²–10⁴ dwarfs cost model 2× and search 1.2×, graph-JOB design exercise), experiments crate compiles: toy cost-based planner todo!() stubs (parse_and_plan naive left-deep → push_down → greedy reorder_joins → estimate with 1/NDV + independence + containment) with contract tests incl. join_order_flips_with_stats, explain binary PROVIDED for side-by-side DuckDB EXPLAIN comparison.
• 2026-07-10 — topic 9 scaffolded: study guide (latch vs lock table, memory-ordering cheat sheet + publication idiom, latch-coupling→OLC→lock-free ladder, epoch reclamation diagram, Bw-tree cautionary arc, false sharing), 4 reading guides (postgres lwlock.c packed u32 + recheck-after-enqueue lost-wakeup dance; crossbeam-epoch pin/defer/try_advance — newly cloned; RocksDB InlineSkipList CAS+splices vs memgraph lazy-locking skiplist with accessor-id GC — memgraph newly cloned; Bw-tree ICDE’13 + SIGMOD’18 reality check + Leis OLC), experiments crate compiles: lock-free ConcurrentSet todo!() stub over crossbeam-epoch with 5 contract tests (same-key/remove races exactly-one-winner, reader-survives-removal-churn UAF canary), scaling shootout PROVIDED (global mutex / 16-shard / crossbeam SkipSet / yours, 1→16 threads), false_sharing PROVIDED and run — packed 63 M inc/s vs pad128 3707 M (59×), and pad64 still 2.2× slower than pad128: Apple M-series coherence granularity is 128 B, x86-style 64 B padding only half-fixes it.
• 2026-07-10 — topic 8 scaffolded: study guide (anomaly-per-isolation-level table, doctors write-skew walkthrough, 2PL/OCC/MVCC comparison, postgres tuple-header + visibility flowchart, HOT chain, Hekaton contrast), 6 reading guides (postgres heapam.c/heapam_visibility.c HeapTupleSatisfiesMVCC + HOT + prune/vacuum with line anchors; RocksDB optimistic vs pessimistic txns over one base class — memtable-only OCC validation, point lock manager; surrealdb kvs layer — newly cloned — versioned reads + putc as portable OCC; Berenson ’95 history notation + SI dethroned; SSI VLDB’12 dangerous structure + the single-writer M8 shortcut question; Hekaton + Wu/Pavlo 5-axis menu), experiments crate compiles: Mvcc todo!() stub with 8 contract tests including write_skew_HAPPENS_under_SI (test passes when the anomaly occurs) and Serializable-mode prevention via read-set validation, txn_bench PROVIDED (global Mutex baseline vs MVCC, 3 mixes incl. 64-key hot set, abort counts).
• 2026-07-10 — topic 7 scaffolded: study guide (RESP wire anatomy, event-loop mermaid beforeSleep→poll→read→execute→buffer, three threading models table, backpressure: querybuf/output-buffer kills vs pgwire portals), 4 reading guides (redis ae.c + networking.c parse/reply path with line anchors; valkey 8 io_threads.c SPSC inboxes + tagged job pointers + memory_prefetch.c batch-MLP; pgwire Parse/Bind/Execute/Sync portals + qdrant dual tonic servers — both newly cloned; C10K → thread-per-core arc with the shared↔sharded plane exercise), experiments crate compiles: RESP2 parse/encode todo!() stub with 8 format-fixing tests (incomplete-input-keeps-bytes, binary-safe bulks, pipelining), tokio server PROVIDED (16-shard store, parse-all-then-flush-once pending-writes trick) — benches vs real redis via redis-benchmark -P 1/-P 64 + flamegraph once resp.rs is implemented.
• 2026-07-10 — topic 6 scaffolded: study guide (translation-cost table hash/swizzle/MMU, miss-path mermaid, three shapes of approximate-LRU, swip state diagram, mmap CIDR-’22 checklist), 6 reading guides (postgres bufmgr.c packed-atomic state + CLOCK + buffer rings; DuckDB eviction queue with dead nodes + 4096-insert purge — newly cloned; LeanStore swips/cooling/hybrid latches — newly cloned; redis zmalloc per-thread padded counters + turso CLOCK page cache bonus; mmap paper with LMDB rebuttal; LeanStore+vmcache paper arc), experiments crate compiles: CLOCK BufferPool todo!() stub with contract tests (pinned-never-evicted, dirty-writeback, scan-pressure survival), pool_vs_mmap binary (1GiB file, 4× memory budget, Zipf, tail-latency focus), eviction bench PROVIDED and run — CLOCK 67.0% vs strict-LRU 66.3% hit rate at 20× less time per access (32ms vs 678ms per 1M trace): the “nobody ships strict LRU” lesson, measured.
• 2026-07-10 — topic 5 scaffolded: study guide (WAL rule, four-designs axis LMDB→turso→postgres→redis-AOF, fsync ladder table, group-commit mermaid), 5 reading guides (postgres xlog.c — newly cloned — reserve-then-copy/XLogFlush-recheck/FPI with line anchors; turso WAL checksum chain + salts; redis aof.c/rdb.c with the AOF-as-LSM mapping + FalkorDB angle; ARIES three passes/CLRs; Aether four-bottleneck taxonomy), experiments crate compiles: fsync_ladder PROVIDED and run (this Mac: fsync 21µs vs F_FULLFSYNC 3.0ms — 140×, the macOS weak-fsync gap is real), Wal todo!() stub with format-fixing tests (torn tail, uncommitted-txn invisibility, commit_many = 1 fsync), crash_test kill-9 harness (100 rounds, acked-key + atomicity checks), commit_throughput bench (per-commit vs group 8/64/512).
• 2026-07-10 — topic 4 scaffolded: study guide (memtable→SST lifecycle mermaid, SST block anatomy, leveled/tiered/lazy RUM table, stall triggers, Monkey intuition), 6 reading guides (lsm-tree crate — newly cloned, fjall delegates to it — + RocksDB compaction/table with line anchors; Monkey, Dostoevsky, RocksDB TODS ’21, compaction design-space VLDB ’21), experiments crate compiles: mini-LSM with provided Bloom (tests pass) + Memtable, SST writer/reader + Lsm engine todo!() stubs with correctness tests (tombstone-across-compaction, WA>1 check), write_amp binary measuring the full RUM position of leveled vs tiered.
• 2026-07-10 — topic 3 scaffolded: study guide (slotted page anatomy, 3-sibling balance mermaid, LMDB double-meta COW commit diagram), 5 reading guides (turso btree deep + SQLite btree.c + LMDB mdb.c with line anchors from fresh clones; Graefe survey selective-read map, SQLite file-format hex-dump exercise), experiments crate compiles: slotted Page + DiskBTree todo!() stubs with format-fixing tests, bench vs redb (point/scan) + prefix-truncation stress case (32B keys, 24B shared prefix).
• 2026-07-10 — topic 2 scaffolded: study guide (chaining vs open addressing cache stories, incremental-rehash mermaid, skiplist/rax ASCII, dense-filter/fat-payload pattern table), 7 reading guides (redis dict/zset/rax, hashbrown SwissTable, RocksDB InlineSkipList — line numbers from local clones; ART paper, CppCon SwissTable talk), experiments crate compiles: skiplist + incremental_map todo!() stubs with tests (the build work is the learning work), benches vs hashbrown/BTreeMap/crossbeam-skiplist, rehash_spike binary (HdrHistogram per-insert max/p99.9).
• 2026-07-10 — topic 1 started: study guide (two-family write/read paths, amplification vocabulary, RUM triangle), 8 reading guides (fjall/turso/tidesdb/rocksdb code + O’Neil/Comer/RUM/Hellerstein papers, line numbers from fresh shallow clones), engine_shootout scaffold (fjall vs redb behind a common trait, db_bench workload names, durability parity) compiles + smoke-tested (space-amp binary at 20K keys shows fixed-overhead floor, not amplification — re-run at 1M+). Topic 0 plan audit: fixed phantom CMU-lecture reference in PLAN.md, added missing roofline-thinking section to topic 0 README §4.
• 2026-07-10 — topic 0 finished: cache_ladder (after fixing a self-caching bug: restarting the pointer chase at 0 measured an 8MB hot path — fixed by carrying the walker across iterations; true ladder 1.0 ns L1 / 5–9 ns L2 / ~110 ns DRAM+TLB), lookup_shootout (HashMap flat at ~7–9 ns thanks to MLP; binary search wins ≤1e4; linear scan never beats hashing at n≥100 — folklore busted), flamegraph captured (21% SipHash in HashMap lookups), reference baselines recorded in capstone/BASELINES.md. Topic 0 + M0 done.
• 2026-07-10 — topic 0 started: study guide + 3 experiment benches (cache_ladder, lookup_shootout, branch_misprediction); capstone workspace scaffolded with workload crate (seeded Zipfian generator, ~11M ops/s). First measured result: branchy filter 8.1x slower on shuffled vs sorted data; branchless flat at 15 Gelem/s. Repo published to github.com/AviAvni/database-learning-path.
• 2026-07-10 — repo initialized: plan, capstone design, resources.

Topic 0 — The Performance Toolbox

Learn to measure before learning to build. Every topic after this one ends with a benchmark; this topic makes sure those benchmarks tell the truth.

Outcomes

By the end you can:

1. Write a microbenchmark that isn’t lying to you, and explain why it isn’t.
2. Read a flamegraph and perf stat-style counters and name the bottleneck (CPU-bound, memory-bound, branch-miss-bound, syscall-bound).
3. Recite the memory-hierarchy latency ladder from registers to disk within 2x.
4. Report latency correctly (percentiles, not means) and explain coordinated omission.

1. Why microbenchmarks lie

The five classic failure modes — you will hit all of them this week on purpose:

• Dead-code elimination: the optimizer deletes your benchmarked work because the result is unused. Fix: std::hint::black_box on inputs and outputs.
• Warmup & frequency scaling: first iterations pay cold caches, page faults, and low CPU clocks. Criterion warms up for 3s by default — that’s not superstition.
• Variance & noise: background processes, turbo boost, thermal throttling. Criterion runs many samples and does outlier detection; single-shot timing is fiction. On Apple Silicon: P-cores vs E-cores — pin expectations, not threads (macOS gives you no affinity API; keep the machine idle instead).
• Wrong distribution: benchmarking uniform random keys when production is Zipfian (redis workloads are heavily skewed). The workload generator you build in M0 exists to fix this permanently.
• Coordinated omission: if you measure latency by sending request → wait → send next, a stall backs up your load generator and the stall disappears from your data. Gil Tene’s “How NOT to Measure Latency” talk is mandatory. This is why redis-benchmark got --latency modes and why HdrHistogram exists.

Coordinated omission, on a timeline (each | is one request):

intended schedule   | | | | | | | | | | | | | | | | | |     constant rate
actual (closed loop)| | | | | |. . . . . . . .| | | | |     generator waits with the server
                               ^--- server stall ---^
what you record:    fast fast fast [ONE slow sample] fast    → "p99 looks great!"
what users felt:    every request due in the stall window waited up to the full stall

Fix: timestamp each request with its INTENDED send time; latency = completion − intended.

2. The memory hierarchy — the numbers that explain everything

Approximate, Apple M-series / modern x86, per access:

The same ladder on a log scale — each step down is roughly an order of magnitude:

               1ns    10ns   100ns    1µs    10µs   100µs    1ms
L1     ~1 ns   ●
L2     ~4 ns   ──●
SLC   ~15 ns   ────●
DRAM  ~90 ns   ───────●
NVMe  ~50 µs   ──────────────────────────●
fsync ~0.5 ms  ────────────────────────────────────────●
               └─ one DRAM miss ≈ 100 adds; one fsync ≈ 10,000 DRAM misses

flowchart LR
    CPU["CPU core<br/>registers"] --> L1["L1<br/>64–192 KB · ~1 ns"]
    L1 --> L2["L2<br/>4–16 MB · ~4 ns"]
    L2 --> SLC["L3 / SLC<br/>8–96 MB · ~15 ns"]
    SLC --> DRAM["DRAM<br/>GBs · ~90 ns"]
    DRAM --> NVMe["NVMe<br/>TBs · ~50 µs"]
    NVMe --> FSYNC["fsync<br/>~0.5 ms"]

Consequences you’ll verify in the experiments:

• A DRAM miss costs ~100 sequential integer additions. Databases are mostly memory-hierarchy management — B-trees (page-sized nodes), LSM SSTs (sequential IO), vectorized execution (cache-resident batches), roaring bitmaps (fit in cache) are all answers to this table.
• Sequential beats random not because of “seek time” (that’s spinning disks) but because of prefetching and cache-line granularity: touching 1 byte loads 64–128 bytes.
• Pointer chasing (linked lists, naive trees, neo4j-style record hopping) serializes misses: each next address depends on the previous load. Arrays let the CPU overlap many misses (memory-level parallelism). This single fact is why FalkorDB’s matrix-based adjacency can beat pointer-based graph stores.

Why arrays beat pointer chasing — memory-level parallelism:

pointer chase (next address depends on previous load — misses serialize):
  load A ═══════►
                 load B ═══════►
                                load C ═══════►        3 × ~100 ns = ~300 ns

array scan (addresses known up front — misses overlap):
  load A ═══════►
  load B  ═══════►
  load C   ═══════►                                    ≈ ~110 ns total

3. Branches and the pipeline

Modern cores are ~8-wide and ~500 instructions deep in flight. A mispredicted branch flushes the pipeline: ~15–20 cycles. A 50%-unpredictable branch in a per-row filter is catastrophic; the fix is branchless/SIMD selection (topic 17). You’ll measure the sorted-vs-unsorted filter effect in experiment 3 — the classic 5–10x gap.

per-row filter `if x > t`, 50% unpredictable:

predict OK :  [fetch|decode|exec|...]───►  next row overlaps, ~1 row/cycle
mispredict :  [fetch|decode|exec|✗ FLUSH]  ~15–20 cycles of speculative work discarded
                                  └──► restart fetch from the correct path

branchless :  sum += (x > t) as u64 * x    no control dependence — nothing to predict

4. Tools on this machine (macOS ARM)

Install: cargo install samply flamegraph hyperfine; brew install fio

Habit to build: criterion tells you what got slower, the profiler tells you why, counters tell you why at the hardware level. Always use at least two of the three before believing a conclusion.

Roofline thinking — before optimizing a kernel, ask which wall it’s against. Peak performance is bounded by min(peak compute, bandwidth × arithmetic intensity), where arithmetic intensity = ops per byte moved. Low intensity (scans, hash probes: ~0.1–1 op/byte) → memory-bound: more SIMD won’t help, cache-friendly layout will. High intensity (compression, hashing wide rows) → compute-bound: SIMD/algorithm wins.

 perf
  │            peak compute ────────────────
  │           ╱
  │          ╱ ← bandwidth roof (slope = GB/s)
  │         ╱
  │        ╱
  │   scan●        ●hash probe        ●compression
  └───────┴────────┴─────────────────┴──────────── ops/byte
        memory-bound ◄─── ridge point ───► compute-bound

Most database kernels live left of the ridge — that’s why this whole curriculum is obsessed with the memory hierarchy rather than FLOPs.

flowchart LR
    C["criterion<br/>WHAT got slower<br/>(numbers + CIs)"] --> P["samply / flamegraph<br/>WHY — which code<br/>(where time goes)"]
    P --> H["CPU counters<br/>WHY — which hardware limit<br/>(cache/branch/IPC)"]
    H --> C
    style C fill:#1f6feb,color:#fff
    style P fill:#8957e5,color:#fff
    style H fill:#bf4b8a,color:#fff

5. Code reading (2–4 h)

• criterion: read analysis/mod.rs + how it uses warmup and outlier classification. Question to answer: why does it report a confidence interval rather than a minimum? → chapter: reading-criterion.md — How criterion turns noise into a number you can trust
• redis redis-benchmark.c: how does it implement pipelining? What does it get wrong about coordinated omission? → chapter: reading-redis-benchmark.md — redis-benchmark: a throughput tool wearing latency clothes
• RocksDB tools/db_bench_tool.cc (skim): note the workload flags — this is the vocabulary of storage benchmarking (fillseq, readrandom, overwrite…). → chapter: reading-rocksdb-db-bench.md — db_bench: the shared vocabulary of storage benchmarking

6. Reading / watching

• Brendan Gregg, Systems Performance ch. 1–2 (methodologies — USE method).
• Gil Tene, “How NOT to Measure Latency” (talk) — lessons captured in notes.md.
• Raasveldt, Holanda, Gubner, Mühleisen — “Fair Benchmarking Considered Difficult: Common Pitfalls in Database Performance Testing” (DBTest ’18, from the DuckDB authors) — the DB-specific companion to this topic’s §1. → chapter: reading-fair-benchmarking.md — Fair benchmarking: eight ways a system comparison lies
• “What Every Programmer Should Know About Memory” (Drepper) — §3–4 only, skim the rest. → chapter: reading-drepper.md — CPU caches and TLBs: the constants aged, the structure didn’t

7. Experiments (in experiments/)

1. cache_ladder — stride through arrays of 16KB → 512MB; plot ns/access. You should see L1/L2/SLC/DRAM as plateaus.
2. lookup_shootout — point lookups: Vec linear scan vs Vec binary search vs HashMap vs BTreeMap, sizes 1e2 → 1e7. Find the crossover where linear scan beats hashing (it exists, and it’s bigger than you think).
3. branch_misprediction — sum elements > threshold over sorted vs shuffled data. Then make it branchless and watch the gap vanish.
4. Profile experiment 2 with samply; grab one flamegraph screenshot into notes.md.

8. Capstone milestone M0 (in ../../capstone/)

• Cargo workspace scaffolded
• workload crate: seeded, Zipfian-skewed graph workload generator (node/edge inserts, point reads, 2-hop traversal patterns)
• criterion harness wired up (generator smoke bench)
• Baseline numbers from the reference falkordb-rs-next-gen recorded in capstone/BASELINES.md (run its bench suite; numbers to chase later)

Done when

• All three experiments run and their results are explained in notes.md (numbers + why), the flamegraph screenshot is captured, and M0 checklist is complete.

How criterion turns noise into a number you can trust

Every benchmark in this curriculum runs through criterion, so before trusting any of them it pays to know exactly what the tool does to raw timings. The answer lives in one 370-line file, analysis/mod.rs: common() (line 39) is a pipeline, and every line criterion prints during a bench run maps to a specific step here. Three ideas do all the work — bootstrap instead of normality assumptions, slope instead of mean, label outliers instead of dropping them.

The pipeline in common()

flowchart TD
    S["1 · routine.sample()  (line 83)<br/>(iters, times) — iters grows [d, 2d, 3d, ...]"]
    S --> N["2 · avg_times[i] = times[i] / iters[i]  (124–129)"]
    N --> T["3 · tukey::classify  (141)<br/>label outliers — NEVER remove"]
    N --> E["4a · estimates()  (300)<br/>mean/median/MAD, each bootstrapped (321)"]
    S --> R["4b · regression()  (269)<br/>slope of total_time vs iters = ns/iter"]
    R --> H["headline:  time: [lo mid hi]  = slope's bootstrap CI"]
    E --> C["5 · compare.rs  (188)<br/>bootstrapped t-test vs saved baseline<br/>+ noise_threshold gate"]
    H --> C

1. Sampling (line 83) — routine.sample(...) returns (sampling_mode, iters, times): two parallel arrays, e.g. 100 samples where sample i ran iters[i] iterations and took times[i] total. Key trick: it never times a single iteration (too noisy, clock granularity). With linear sampling, iters grows as [d, 2d, 3d, ...] — that shape matters for step 4. Warmup lives in routine.rs::warm_up, before this — its real job is calibrating how big d must be to fill the target time; cold-cache mitigation is a side effect.

2. Normalize (lines 124–129) — avg_times[i] = times[i] / iters[i] → per-iteration averages. This is the Sample all univariate stats run on.

3. Outlier classification (line 141) — tukey::classify labels each sample using Tukey fences (quartiles ± 1.5×IQR / 3×IQR → mild/severe). Outliers are labeled and reported, never removed — that’s the “Found N outliers among 100 measurements” line. Philosophy: deleting data you don’t like is how benchmarks lie.

4. Two independent estimators:

• estimates() (line 300): mean/median/std-dev/MAD of avg_times, each bootstrapped (line 321): resample the sample with replacement nresamples (100k) times, recompute the stat each time → an empirical distribution of the statistic → read the CI off its percentiles. No normality assumption — which matters because latency isn’t normal.
• regression() (line 269): fits total_time = slope × iters through the (iters, times) points — the slope is ns/iteration. Only valid for linear sampling (checked at line 152). The headline time: [lo mid hi] is the slope’s bootstrap CI. Why slope beats mean-of-averages: constant per-sample overhead (measurement, loop setup) inflates every avg_times[i] but shows up as an intercept the slope ignores.

total_time                                 why slope beats mean-of-averages:
    │                          ×
    │                    ×                 slope  = ns per iteration  ← the answer
    │              ×
    │        ×
    │  ×
    ├─────────────────────────── iters
    ╵← intercept = fixed per-sample overhead
       (mean of averages absorbs it; the slope ignores it)

The bootstrap itself — the engine under every CI criterion prints — fits in ten lines:

#![allow(unused)]
fn main() {
fn bootstrap_ci(sample: &[f64], nresamples: usize) -> (f64, f64) {
    let n = sample.len();
    let mut stats = Vec::with_capacity(nresamples);
    for _ in 0..nresamples {                       // 100_000 in criterion
        // resample WITH replacement, same size — pretend the sample IS the population
        let stat = mean((0..n).map(|_| sample[rand_below(n)]));
        stats.push(stat);                          // distribution OF THE STATISTIC
    }
    stats.sort_by(|a, b| a.partial_cmp(b).unwrap());
    (percentile(&stats, 2.5), percentile(&stats, 97.5))  // CI = its percentiles —
}                                                        // no normality assumed
}

5. Regression detection (line 188 → compare.rs) — loads the saved baseline, bootstraps a two-sample t-test (line 200: p_value) asking “is the difference real?”, then a bootstrapped relative-change estimate against noise_threshold asking “is it big enough to care?”. Both gates must pass — e.g. +3781% (p = 0.00 < 0.05): significant and large.

Study-guide question: why a CI, not a minimum?

The “report the minimum” school (older Python timeit advice) argues noise is strictly additive, so min = true cost. Criterion rejects that:

1. Min answers the wrong question. It estimates best-case-ever (all caches hot, zero interference) — a state production code never runs in. The CI estimates typical cost with honest uncertainty bounds.
2. Noise isn’t strictly additive. Frequency scaling means early samples can run at a higher clock (pre-thermal-throttle) — the min can be an unrepresentatively lucky sample, and on modern laptops often is.
3. Min is statistically fragile for comparison. It’s an extreme-value statistic with no usable sampling distribution — you can’t compute a p-value on “min got 2% slower.” The regression-detection machinery only works because mean/slope have bootstrap distributions.
4. A point estimate hides confidence. [69.6 70.1 70.5] µs says the measurement is tight; a bare 69.6 hides whether the spread was 1% or 40%.

Suggested reading order in the crate

1. analysis/mod.rs::common — the spine
2. stats/univariate/outliers/tukey.rs — fences, ~100 lines
3. stats/bivariate/regression.rs — Slope::fit is ~10 lines of least-squares
4. analysis/compare.rs + stats/univariate/mixed.rs — the bootstrapped t-test
5. routine.rs::warm_up — see that warmup is really iteration-count calibration

Takeaway

Criterion is built on three ideas: bootstrap instead of normality assumptions, slope instead of mean, label outliers instead of dropping them.

References

Code

• criterion.rs src/analysis/mod.rs (locally: ~/.cargo/registry/src/index.crates.io-*/criterion-0.5.1/src/analysis/mod.rs, 370 lines) — common() is the spine; follow the suggested reading order above through tukey.rs, regression.rs, compare.rs, and routine.rs::warm_up

CPU caches and TLBs: the constants aged, the structure didn’t

Every latency table in topic 0 §2 is a compressed version of one 2007 paper — Drepper’s “What Every Programmer Should Know About Memory”. This chapter is a reading lens for its two load-bearing sections: §3 (why misses cost what they cost) and §4 (why a TLB miss is pointer chasing in silicon). The DDR2 numbers are stale; the cache-organization math, the prefetching rules, and the measurement methodology behind cache_ladder are forever.

What’s stale vs. what’s forever

2007 paper: the constants aged, the structure didn’t. Reading lens:

• Stale: DDR2 timings, front-side bus, Pentium 4/NetBurst details, exact cache sizes.
• Forever: cache organization math, why misses cost what they cost, prefetching rules, the measurement methodology (his benchmark plots are the blueprint for cache_ladder).
• Apple Silicon deltas to keep in mind while reading: 128-byte cache lines on M-series (not 64!), no inclusive L3 (shared SLC instead), much larger L1 (128–192 KB).

§3 — CPU caches (the core, ~35 pages)

• 3.1–3.2 Skim. Cache hierarchy diagrams + associativity. Know: set-associative = hash table with N-way buckets; conflict misses = bucket collisions.
• 3.3 (read carefully) — the famous measurements. Fig 3.4 (sequential vs random access over working-set size) is exactly the cache_ladder experiment; compare his plateau shapes with yours before explaining your numbers in notes.md. Understand why random is worse than sequential even in DRAM: TLB misses + no prefetch + row activation.
• 3.3.2 Critical word first / early restart — why the miss cost isn’t a full line transfer.
• 3.4 Instruction cache — skim (matters again at topic 19, JIT).
• 3.5 (read carefully) Cache coherency + false sharing (Fig 3.27-ish, multi-thread scaling collapse). This is the section that pays off in topic 9 (concurrency) — two atomics on one line = cacheline ping-pong.
• Fig 3.11 (cache-line utilization) explains why columnar layouts win: touching 8 bytes of a 128-byte line wastes 94% of the transfer. Topic 12 in one figure.

filter on one 8-byte column, 128 B cache lines (M-series):

row layout:  line = [ a │ b  c  d  e  f  g ... padding ... ]   use 8 B / 128 B → 94% wasted
col layout:  line = [ a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a ]   use 128 B → 0% wasted

The measurement engine behind Fig 3.4 (and behind cache_ladder) is a pointer chase through a shuffled ring — every load depends on the previous one, so latency can’t hide behind memory-level parallelism:

#![allow(unused)]
fn main() {
// ring[i] holds the index of the next element to visit (a shuffled cycle).
// Because address N+1 is unknown until load N retires, ns/step == the raw
// latency of whatever level the working set lands in — L1, L2, SLC, DRAM.
fn chase(ring: &[usize], steps: usize) -> usize {
    let mut i = 0;
    for _ in 0..steps {
        i = ring[i];            // serialized miss: nothing to prefetch
    }
    i                           // return it so the loop isn't dead code
}
// grow ring.len() from 16 KB to 512 MB and plot ns/step → the plateaus
}

§4 — Virtual memory (~10 pages)

• 4.1–4.2 Page tables are a 4-level radix tree walked in memory — a TLB miss is up to 4 dependent loads. Sound familiar? It’s pointer chasing (topic 0 §2).

A TLB miss is pointer chasing in silicon — 4 dependent memory loads:

CR3 ──► PGD entry ──► PUD entry ──► PMD entry ──► PTE ──► finally, your data
        (load 1)      (load 2)      (load 3)     (load 4)
        each load can itself miss cache ⇒ worst case ~4 × DRAM latency
        before the ACTUAL access even starts

• 4.3 (the key bit) TLB reach: 4 KB pages × ~2K entries ≈ a few MB — far smaller than working sets. Why databases care about huge pages (2 MB/1 GB; 16 KB base pages on Apple Silicon already 4x the reach).
• Skim the virtualization part (4.4+).

§6 — What programmers can do (skim for the checklist)

Sequential access > random; -O2 -march=native; struct layout: hot fields together, sorted by size; pahole-style padding audits; NUMA awareness (§5/§7 — skip until a NUMA box matters). §6.2’s cache-oblivious matrix transpose is worth 10 minutes — it’s the intellectual ancestor of blocked/vectorized execution (topic 11).

Questions to answer in notes.md when done

1. Why does cache_ladder show gradual transitions between plateaus rather than steps? (Hint: set associativity + random chain touching multiple sets.)
2. Predict: on M-series with 128 B lines, at what stride does a strided-read benchmark stop getting faster per element? Verify with a quick experiment.
3. How many memory accesses can a single TLB miss add on a 4-level page table, and why don’t we see it in cache_ladder? (Hint: 16 KB pages, working set vs TLB reach.)

Takeaway

Every table in topic 0 §2 is a compressed version of this paper. Drepper’s method — plot access cost against working-set size and explain every inflection — is the habit; the numbers you regenerate yourself on your own machine.

References

Papers

• Drepper — “What Every Programmer Should Know About Memory” (Red Hat, 2007) — PDF (~114 pages — read §3–§4 properly, skim §6, skip the rest; the study guide’s advice stands)

Fair benchmarking: eight ways a system comparison lies

Criterion and Tene cover how a single measurement lies; this chapter — built on a 6-page DBTest ’18 paper from the future DuckDB authors — covers how a comparison between systems lies. It is the database-specific companion to topic 0 §1, and its Appendix A checklist is an artifact you will reuse against every capstone comparison in this curriculum.

Structure

• §1–2 Intro + related work (skim, but note the gems): Jain’s mistakes vs games distinction; Hoefler & Belli’s 12 HPC benchmarking rules; van der Kouwe’s survey finding benchmarking crimes in 96% of 50 top-tier systems papers; Purohith et al.: SQLite throughput varies 28x on one parameter, and 0 of 16 surveyed papers reported it.
• §3 The eight pitfalls, each with a mock TPC-H SF1 experiment (MariaDB / PostgreSQL / SQLite / MonetDB, single-threaded) — this is the part to read carefully.
• §4 + Appendix A Conclusions + the checklist (the artifact you’ll reuse).

The eight pitfalls (§3)

flowchart TD
    Q["Where a system comparison lies"]
    Q --> SU["Setup"]
    Q --> CMP["Comparison"]
    Q --> MEA["Measurement"]
    Q --> RES["Results"]
    SU --> P1["3.1 non-reproducible<br/>(the Escher result)"]
    SU --> P2["3.2 untuned baseline<br/>(debug build, default config)"]
    CMP --> P3["3.3 apples vs oranges<br/>(kernel vs full system)"]
    CMP --> P4["3.4 tuned to the benchmark<br/>(known selectivities)"]
    MEA --> P5["3.5 cold/hot conflated"]
    MEA --> P6["3.6 restart ≠ cold<br/>(OS page cache warm)"]
    MEA --> P7["3.7 preprocessing ignored<br/>(index build, auto-imprints)"]
    RES --> P8["3.8 fast but wrong<br/>(diff against a trusted engine)"]

1. Non-reproducibility (3.1) — the Escher result: Fig. 2 shows MariaDB < Postgres < SQLite < MariaDB*, all “true”. The trick: MariaDB* used DOUBLE instead of DECIMAL columns — both allowed by the TPC-H spec, invisible unless the full setup is published.
2. Failure to optimize (3.2) — the baseline is the author’s competitor, so nobody tunes it. MonetDB debug build vs release: 1.58s → 0.87s (Q1). Postgres default vs configured: 0.47s → 0.27s (Q9). “DBMS A vs B” can be the same system twice.
3. Apples vs oranges (3.3) — hand-written Q1 program (“TimDB”) vs MonetDB: 0.03s vs 0.87s. A standalone kernel skips parsing, transactions, overflow checking, concurrency. Compare full system vs full system, and verify identical results.
4. Overly-specific tuning (3.4) — TPC-H’s selectivities/cardinalities are known, so join heuristics can be tuned to the benchmark. Antidote: also run non-benchmark queries.
5. Cold vs hot runs (3.5) — report them separately; hot runs discard initial iterations (criterion’s warmup, formalized).
6. Cold vs warm runs (3.6) — subtler: restarting the server is NOT a cold run; the OS page cache is still warm. True cold = stop server, echo 3 > /proc/sys/vm/drop_caches, start, one query, repeat. (Nearly impossible in cloud — the hypervisor caches too.)
7. Ignoring preprocessing time (3.7) — excluding index-build time rewards expensive-to-build indexes. Watch automatic preprocessing: MonetDB builds imprints on first range filter and dictionary-encodes strings at load — “cold” first-query timing silently includes/excludes work per system.
8. Incorrect code (3.8) — a fast wrong answer wins benchmarks (skipped overflow handling, hardcoded group counts). Always diff results against a trusted engine.

Methodology to steal (§3 preamble)

Their own reporting standard: median + non-parametric quantile-based 95% confidence intervals, all scripts/configs/plots public. Same philosophy as criterion (§ bootstrap CIs), applied to system-level runs.

Connections to this repo

• The capstone’s M4 backend shootout and M22 LDBC 3-way FalkorDB comparison must pass Appendix A — especially 3.2 (tune the reference FalkorDB properly) and 3.3 (a young engine missing features is structurally “TimDB” — say so explicitly next to numbers).
• FalkorDB/benchmark audit overlaps: no warmup (3.5), timeout asymmetry (3.3-ish), uniform keys (3.4’s cousin — tuning the workload to flatter caches).
• 3.7 is why M0’s workload crate measures generation throughput separately from engine time.

Questions to answer in notes.md

1. Which Appendix A checklist items does FalkorDB/benchmark currently fail? (I count at least four — list them.)
2. The paper reports medians + CIs; Tene demands full percentile curves + max. When is each right? (Hint: throughput-style repeated identical runs vs latency under load.)
3. Which “automatic preprocessing” (3.7) exists in FalkorDB that a fair Neo4j comparison must account for?

Takeaway

Appendix A is a reusable review checklist: benchmarks chosen + justified; reproducible (hardware, params, code, data); both systems optimized; same functionality; cold/hot separated and correctly collected; preprocessing equalized; results verified; medians + CIs over several runs. Pin it next to every capstone notes.md comparison.

References

Papers

• Raasveldt, Holanda, Gubner, Mühleisen — “Fair Benchmarking Considered Difficult: Common Pitfalls in Database Performance Testing” (DBTest 2018) — PDF — 6 pages, one evening; read §3 carefully, Appendix A is the reusable artifact. (CWI — Raasveldt & Mühleisen later created DuckDB.)

Code

• pholanda/FairBenchmarking — the paper’s experiment scripts and configs

redis-benchmark: a throughput tool wearing latency clothes

The load generator you’ll imitate — and the mistake you’ll avoid. In one dependency-free file, redis-benchmark shows a masterclass in cheap pipelining (one pre-built buffer, patched in place) and, in the same 2000 lines, the canonical case of coordinated omission: a closed loop that measures service time and calls it latency. Two questions drive the read: how does it implement pipelining, and what does it get wrong about coordinated omission?

Structure

How pipelining works (the elegant part)

c->obuf — the whole benchmark is one pre-built buffer, written over and over:

┌──────┬────────┬──────────────────────────┬──────────────────────────┬─ ─ ─
│ AUTH │ SELECT │ SET key:__0000000042__ v │ SET key:__0000000913__ v │ ×pipeline
└──────┴────────┴──────────▲───────────────┴──────────▲───────────────┴─ ─ ─
  trimmed after 1st reply  └── randptr[] patch digits in place — no re-serialization

There is no request queue. createClient (625) copies the same command bytes config.pipeline times into one output buffer c->obuf, sets c->pending = config.pipeline, and the event loop just writes the whole buffer and counts replies back down (readHandler, 458: while(c->pending)). Randomized keys are patched in place through saved pointers into the buffer (randptr, 377–393 — writes digits directly into the command bytes, no re-serialization). Auth/SELECT prefix commands ride in the same buffer once and are trimmed after the first reply (506–523).

Cost of the trick: within one batch every pipelined command has the same key randomization per slot of the buffer, and the whole batch is one timing unit.

Where the latency numbers come from

• writeHandler 574: c->start = ustime() when a batch begins writing.
• readHandler 452: if (c->latency < 0) c->latency = ustime() - c->start — on the first read event only. So “latency” = batch send → first bytes of first reply. Deliberate (the comment says parsing overhead shouldn’t count), but it means the last reply’s extra wait is invisible.
• 528–541: that single value is recorded into the HdrHistogram once per reply — all pipeline requests inherit the first reply’s latency. With -P 100, one measurement pretends to be 100.

What it gets wrong about coordinated omission

flowchart LR
    W["writeHandler (555)<br/>c->start = ustime()"] --> R["readHandler (442)<br/>latency = now − start<br/>on FIRST reply only (452)"]
    R --> D["clientDone (420)"]
    D --> RC["resetClient (368)"]
    RC -->|"next batch starts only after<br/>the previous one finished"| W

The cycle above is the whole problem: it’s closed — there is no intended-arrival schedule anywhere, so a server stall pauses the generator itself. In detail: clientDone (420) → resetClient (368) → re-arm write handler → next batch starts after the previous one finished. c->start is set at send time, not against any intended schedule. Consequences, in Tene’s terms:

1. No target rate exists. The benchmark always sends as fast as the server answers, so a stall (fork for RDB save, AOF fsync, slow command) simply pauses the generator — requests that would have arrived during the stall are never sent, never measured. You get exactly one bad sample per client per stall instead of thousands.
2. It measures service time and calls it latency. Queueing delay a real open-world client would experience never appears.
3. HdrHistogram doesn’t save it. Redis added HdrHistogram (config lines 99–100) and full percentile output (830+) — good display of a biased sample. Correction would require an intended-arrival schedule, which doesn’t exist here. (Compare wrk2, which was written to fix exactly this; memtier_benchmark has --rate-limiting.)
4. Small extra: hdr_record_value clamps at CONFIG_LATENCY_HISTOGRAM_MAX_VALUE (line 530) — the worst outliers are also truncated.

Both loops, distilled to their timing skeletons — the entire bug and the entire fix is where the clock starts:

#![allow(unused)]
fn main() {
// closed loop (redis-benchmark): clock starts at SEND — a server stall
// pauses the generator, so the requests that would have queued up behind
// the stall are never sent, never measured.
loop {
    let start = now();
    send_batch_and_wait_all_replies();
    record(now() - start);              // one bad sample per stall
}

// open loop (the fix): clock starts at the INTENDED send time — the
// schedule advances whether or not the server keeps up.
let mut intended = now();
loop {
    intended += period;                 // target rate exists
    wait_until(intended);
    send_one();                         // reply handled async
    on_reply(move |t| record(t - intended));  // queueing delay is visible
}
}

Takeaway

redis-benchmark is a throughput tool with percentile decoration: buffer-replication pipelining is a masterclass in doing the minimum work per event-loop tick, but the closed loop means its latency numbers systematically flatter the server under stress. For the capstone (M7+): keep the obuf trick, add an intended-send schedule.

References

Code

• redis src/redis-benchmark.c (2028 lines, pinned at Redis 8.6.2 / a176d1225) — one file, no dependencies beyond hiredis + the ae event loop; readable top to bottom in an evening

db_bench: the shared vocabulary of storage benchmarking

fillseq, readrandom, readwhilewriting — these workload names started in LevelDB, were extended by RocksDB, and now appear in every LSM paper since. This chapter is a skim route through the 10,000-line tool that defines them: the goal is the vocabulary and the measurement shape, not the harness code. Name your own benchmarks in this language and your numbers become comparable to two decades of published results.

Skim route (30–60 min)

The vocabulary worth memorizing

Under every workload name sits the same skeleton — pick an integer, format it as a fixed-width key (GenerateKeyFromInt :3802, WriteMode :5869, KeyGenerator :6088):

#![allow(unused)]
fn main() {
// every fill* workload reduces to how the next integer is chosen
fn next_key(mode: WriteMode, i: u64, n: u64, rng: &mut Rng, perm: &[u64]) -> Key {
    let int = match mode {
        WriteMode::Sequential   => i,                // fillseq: in-order, no
                                                     //   compaction debt
        WriteMode::Random       => rng.next() % n,   // fillrandom/overwrite:
                                                     //   duplicates → garbage
                                                     //   → compaction pressure
        WriteMode::UniqueRandom => perm[i as usize], // pre-shuffled permutation:
                                                     //   random order, no dups
    };
    generate_key_from_int(int)                       // zero-padded fixed width
}
}

• fillseq — sequential-order load. Fast path for an LSM (no compaction debt); papers use it to build the DB before the real test.
• fillrandom / overwrite — random inserts vs. random overwrites of existing keys (overwrite creates garbage → compaction pressure — different beast).
• fillsync — one fsync per write, N/1000 ops: measures durability cost, not throughput.
• readrandom / readseq / readreverse — point lookups vs. iterator scans.
• readwhilewriting — 1 writer + N readers: the “does compaction wreck my read tail?” test. The *whilemerging/*whilescanning variants isolate other interference.
• seekrandom — iterator Seek cost (touches every level; very different profile from Get).
• multireadrandom — MultiGet batching.
• mixgraph (4133) — the odd one out: models Facebook’s measured production distributions (the “Characterizing, Modeling…” FAST’20 paper) with two-term-exponential key ranges (keyrange_dist_a..d) and Pareto value sizes. The industrial answer to “uniform random keys are the wrong distribution” — same motivation as the capstone’s Zipfian workload crate.
• Standard invocation shape: db_bench --benchmarks=fillseq,readrandom --num=10000000 --value_size=100 --histogram — comma list runs in order against the same DB, so earlier benchmarks create the state later ones measure. That ordering is the methodology.

What to notice about measurement

flowchart TD
    F["--benchmarks=fillseq,readrandom,--histogram<br/>comma list runs IN ORDER against the same DB"]
    F --> RUN["Benchmark::Run() (4030)<br/>workload name → method pointer"]
    RUN --> RB["RunBenchmark(n, name, method) (4583)<br/>spawn N threads"]
    RB --> T1["thread 1<br/>Stats + HistogramImpl (2436)"]
    RB --> T2["thread 2 ..."]
    RB --> TN["thread N"]
    T1 --> M["merge per-thread histograms (2488)<br/>never average percentiles — Tene's rule"]
    T2 --> M
    TN --> M
    M --> OUT["default output: throughput<br/>latency histogram only with --histogram"]

• Per-op latency goes through FinishedOps (2564) into a plain HistogramImpl — reported only with --histogram. Default output is throughput (ops/s, MB/s).
• It’s a closed loop like redis-benchmark: each thread issues the next op after the previous completes. There’s a --benchmark_write_rate_limit/read-rate variant for paced writes, but no coordinated-omission correction — same critique applies.
• db_bench measures the embedded engine (no network), so “latency” here is service time by construction — legitimate for engine work, misleading if quoted as user latency.

Takeaway

db_bench’s value is the workload taxonomy, not the harness. When topic 4 (LSM) and M4 (backend shootout) arrive, name capstone benches in this vocabulary (fillseq, readrandom, readwhilewriting) so numbers are comparable against published RocksDB results.

References

Papers

• Cao, Dong, Vemuri, Du — “Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook” (FAST 2020) — the measured production distributions behind mixgraph; optional, skim §4-5

Code

• rocksdb tools/db_bench_tool.cc (~10,400 lines, shallow clone @ 7c80a5a) — do not read this linearly; it’s a flag-driven monolith — follow the skim route table above (30–60 min)

Topic 0 — Notes

Numbers from this machine (Apple Silicon, macOS). Record why, not just what.

Talk: Gil Tene — “How NOT to Measure Latency” (watched ✅)

Core thesis: almost everyone measures latency wrong, and the errors all point the same direction — making systems look better than they are.

1. Latency is a distribution, never a number. Means and standard deviations are meaningless for latency (it’s multi-modal, heavy-tailed, not normal). Always report percentiles — and the whole curve, not just p50/p99.
2. The tail is what users experience. A page load touching ~100 resources hits the p99 almost every time (1 − 0.99¹⁰⁰ ≈ 63%). “p99.9 doesn’t matter” is backwards: the more requests per user interaction, the deeper the percentile that dominates UX.
3. Coordinated omission — the big one. If the load generator waits for a response before sending the next request, a server stall silences the generator exactly when things go bad: the bad results are omitted from the data, coordinated with the stall. A 100s test with one 50s pause can report “p99 < 1ms” while reality is ~25s average during half the test. The error is ~1000x+, not a rounding issue.
   • Fix: measure against the intended send schedule (constant-rate arrival), not the actual send time. If a request should have gone out at t=5s but went out at t=55s, its latency includes those 50s of wait.
   • This is why HdrHistogram has correction modes and why wrk2/redis-benchmark grew constant-throughput modes.
4. Service time ≠ response time. Service time = how long the server took once it started; response time = what the client experiences, including queueing. Load generators that back off measure service time and call it response time. Throughput-vs-latency plots made this way are fiction beyond saturation.
5. “Sustainable throughput” framing. Don’t ask “what’s the max throughput?” — ask “what’s the max throughput at which we still meet the latency requirements?” Test by stating requirements first (e.g. p99.9 < 20ms, max < 200ms), then finding the highest load that passes. A benchmark without a latency requirement is a throughput benchmark, and throughput alone is easy to game.
6. Beware the hockey stick you can’t see. Plotted percentile curves always bend up hard somewhere (“the hockey stick”); tests that stop at p99 just hide where. Plot to the max recorded value — the max is a real event that happened, not an outlier to trim.
7. Never average percentiles across intervals/machines — p99s don’t average. Merge the histograms (HdrHistogram), then read percentiles off the merged data.

Rules for this repo’s benchmarks (from the talk):

• Capstone server benches (M7+) must use a constant-rate open-loop load generator with coordinated-omission correction (HdrHistogram), never closed-loop request→wait→request.
• Report p50/p90/p99/p99.9/max + full percentile plot; never mean latency.
• Criterion is fine for CPU microbenches (throughput of kernels), but not an oracle for request latency — different tool for a different question.

Experiment 3 — branch_misprediction (done, first pass)

• 8.1x sorted→shuffled gap for the branchy version — the classic misprediction penalty. 1M elements, ~50% unpredictable taken rate ⇒ ~500K flushes; (2750−338)µs / 500K ≈ ~4.8 ns ≈ 15 cycles per miss at ~3.2 GHz. Matches the §3 estimate.
• Surprise: with plain sum += x in the branch, LLVM if-converts + auto-vectorizes and the gap vanishes (both ~70µs). Had to put black_box(x) inside the taken path to keep a real branch. Lesson: on modern compilers the famous StackOverflow sorted-array effect only reproduces if vectorization is defeated — always check the asm.
• Branchless = data dependence instead of control dependence ⇒ NEON select, 4.8x faster than even the perfectly-predicted branchy loop.

Experiment 2 — lookup_shootout (done)

ns per lookup (median, 1024 shuffled probes, all hits):

• HashMap is almost flat (7→9.3 ns) even at 1e7 — a ~160MB table where a random probe “should” cost a ~100ns DRAM miss. The 1024 probes are independent, so the out-of-order window overlaps many misses (memory-level parallelism — §2’s array-scan lesson applied to hashing). Single dependent lookups would be much slower; batch APIs exist precisely to expose this parallelism.
• Binary search beats HashMap up to n ≈ 1e4 (3.0–8.1 ns): log₂(n) dependent compares, but the top of the tree stays cache-resident. Past 1e5 the deep levels miss and its dependent-load nature (no MLP within one search) makes it grow ~log(n) × miss cost.
• BTreeMap overtakes binary search at 1e7 (38.9 vs 44.1 ns): ~11-key nodes mean fewer distinct cache lines touched than 23 scattered probes of binary_search — page-sized fanout is the whole point of B-trees (topic 3).
• Study-guide claim busted: iter().find linear scan never beats hashing at n ≥ 100 (17.8 vs 7.4 ns at n=100). Early-exit branchy scan averages n/2 compares; the real crossover sits somewhere below n ≈ 32, smaller than folklore says. A branchless/SIMD scan over a tiny array would move it — revisit in topic 17.

Experiment 1 — cache_ladder (done — after fixing a lying benchmark)

First version lied. chase restarted at idx = 0 every criterion iteration, so every iteration re-walked the same 65,536 slots — an ~8MB hot path that fits in L2/SLC no matter how big the array. The “DRAM” plateau read ~25 ns (mostly TLB misses, since those 64K slots spread across the whole 512MB). Textbook §1 failure mode: the benchmark measured cache residency created by the benchmark itself. Fix: carry idx across iterations so the walk keeps visiting fresh lines.

ns per dependent access (median, fixed version):

• Matches the §2 ladder within noise: ~1ns L1, ~5ns L2, ~100ns DRAM. One dependent DRAM access = ~110 sequential adds — verified, not folklore.
• Transitions are gradual, not steps (Drepper’s question 1): random chains straddle levels probabilistically, and set-associativity evicts unevenly.
• The last rise (64MB → 512MB: 87 → 113 ns) is TLB reach, not cache: 512MB / 16KB pages = 32K pages ≫ TLB entries, so most steps add a page-walk on top of the miss.

Flamegraph (done)

Captured from lookup_shootout/hashmap/10000000 via macOS sample → rustfilt → inferno (SVG committed next to these notes; cargo flamegraph’s xctrace path is broken on current Xcode). For interactive profiling use: samply record ./target/release/deps/lookup_shootout-* --bench --profile-time 10 <filter>

• 21% of samples are inside SipHash (core::hash::sip::Hasher::write) — Rust’s default DoS-resistant hasher costs a fifth of total lookup time even on u64 keys at a size (10M) where DRAM misses should dominate. Swapping in FxHashMap/ahash is the obvious first optimization for the capstone’s internal (non-adversarial) maps.
• The remaining ~79% is the inlined probe loop (hashbrown’s SIMD group probe + the memory stalls themselves) — invisible as separate frames because it’s fully inlined into the bench closure. Sampling profilers attribute stalls to the instruction that waits; only counters (topic 0 §4) can split “executing” from “waiting on DRAM”.

M0 workload generator

• Seeded StdRng + rand_distr::Zipf (s=0.99, YCSB default), skewed toward low ids (oldest nodes = hubs, matching preferential attachment).
• Generation throughput: ~11 M ops/s (9.1 ms / 100K ops) — fine for now; if engine benches ever exceed ~10 M ops/s, pre-generate op vectors outside the timed loop.
• Zipf sampling dominates cost (rejection sampling per draw). Alias-table or precomputed CDF is the known fix — note for later, not needed yet.

Topic 1 — Storage Engine Landscape: B-Tree vs LSM

The single most consequential design decision in a database. Every later topic — WAL, buffer pool, MVCC, compaction, columnar layout — is a refinement of the choice made here: update in place, or write out of place?

Outcomes

By the end you can:

1. Draw the write path and read path of both engine families from memory.
2. Predict which family wins a given workload (write-heavy / point-read / scan / space-constrained) before benchmarking, then verify.
3. Explain any measured difference in terms of read/write/space amplification.
4. Recite the RUM conjecture and give one real engine as an example of each corner.

1. The two families

Everything on disk descends from two ideas:

• Page-oriented, in-place (B-tree): the database is a tree of fixed-size pages (4–16KB). Updates find the page and overwrite it. Reads are 1 tree descent. SQLite, Postgres, LMDB, InnoDB, redb.
• Log-structured, out-of-place (LSM): the database is a log. Updates append to a memtable + WAL; background jobs sort and merge immutable runs (SSTs). Reads must check every place a key could hide. RocksDB, LevelDB, Cassandra, fjall, Pebble.

flowchart TB
    subgraph BTREE["B-tree: update in place"]
        W1["write(k,v)"] --> D1["descend tree<br/>(root→leaf, ~3-4 pages)"]
        D1 --> P1["modify leaf page<br/>in buffer pool"]
        P1 --> WAL1["WAL append<br/>(for crash recovery)"]
        P1 -. later, checkpoint .-> DISK1["overwrite page<br/>on disk"]
    end
    subgraph LSM["LSM: write out of place"]
        W2["write(k,v)"] --> WAL2["WAL append"]
        WAL2 --> MT["memtable insert<br/>(sorted, in RAM)"]
        MT -- full --> FLUSH["flush → immutable SST<br/>(sequential write)"]
        FLUSH -.-> COMP["compaction merges SSTs<br/>(background, rewrites data)"]
    end

The read paths mirror the write paths, inverted:

B-tree point read:                     LSM point read:
  root ──► inner ──► leaf                memtable?          ── miss ─┐
  (3-4 page reads, cached               sealed memtables?   ── miss ─┤
   upper levels ⇒ often 1 IO)           L0 SSTs (each one!) ── miss ─┤  bloom filters
                                        L1 SST              ── miss ─┤  exist to skip
                                        L2 SST              ── hit! ─┘  most of these

2. Amplification — the vocabulary of the whole field

For a logical write of B bytes / logical read of one key:

• Write amplification (WA): physical bytes written ÷ logical bytes. B-tree: whole page per dirty record (4KB page / 100B row ⇒ up to 40x, amortized by the buffer pool). LSM: each byte rewritten once per level by compaction (leveled ⇒ ~10x per level fanout… typical WA 10–30x). On SSDs, WA burns endurance and steals bandwidth.
• Read amplification (RA): physical reads ÷ logical reads. B-tree: tree height (~O(log_fanout n), mostly cached). LSM: number of sorted runs to check — memtable + L0 files + one per level; bloom filters cut the misses, not the final hit.
• Space amplification (SA): physical size ÷ logical size. B-tree: fragmentation + ~30% average page slack. LSM: obsolete versions awaiting compaction (tiered can sit at 2x+; leveled ~1.1x).

3. The RUM conjecture

For Read, Update, and Memory (space) overhead: optimizing any two makes the third worse. You can pick where to sit, not escape the triangle.

                    Read-optimal
                        ▲
                       ╱ ╲
                      ╱   ╲          B-tree      → good R, ok U, poor M (slack)
                     ╱  ○  ╲         LSM leveled → good M, ok U, poor R
                    ╱ B-tree╲        LSM tiered  → good U, poor R, poor M
                   ╱         ╲       hash index  → best point-R, no scans
                  ╱ LSM-l     ╲      bitmap/bloom→ M-optimal, approximate R
                 ╱        LSM-t╲
                ▼───────────────▼
        Update-optimal      Memory-optimal

The conjecture’s sharpest claim: engines are not “good” or “bad”, they are positions. Tuning knobs (compaction style, page fill factor, bloom bits/key) move you along the edges continuously. Monkey (topic 4) is literally a Lagrange-multiplier walk on this triangle.

4. Where each family wins

Hybrid reality check: Postgres (B-tree) has a WAL — a log. RocksDB (LSM) has block indexes inside SSTs — little B-trees. The families differ in what is authoritative: the pages, or the log.

5. Code reading (4–6 h)

Read the two Rust engines as protagonists, skim the other two for contrast:

• fjall (~/repos/fjall) — small, clean Rust LSM. Trace insert → journal → memtable → flush, and get → memtable → SSTs. → chapter: reading-fjall.md — fjall: the LSM lifecycle in clean Rust
• turso (~/repos/turso) core/storage/ — SQLite’s B-tree re-implemented in Rust: slotted pages, cursor descent, balance, pager + WAL. → chapter: reading-turso-btree.md — Turso’s B-tree: the canonical page engine, in Rust
• tidesdb (~/repos/tidesdb) — LSM in plain C; nothing hidden behind abstractions. Skim to see memory ordering and disk layout made explicit. → chapter: reading-tidesdb.md — tidesdb: the same LSM with nothing abstracted away
• RocksDB (~/repos/rocksdb) — don’t read it yet; orient in it. Directory map for topic 4 and beyond. → chapter: reading-rocksdb-layout.md — RocksDB: buy the map before walking the territory

6. Papers (4–5 h)

• O’Neil et al., “The Log-Structured Merge-Tree” (1996) — the origin; read for the cost model, skim the component algebra. → chapter: reading-lsm-paper.md — The LSM-tree: an IO scheduling policy, not a data structure
• Comer, “The Ubiquitous B-Tree” (1979) — still the cleanest B-tree intro ever written. → chapter: reading-comer-btree.md — The B-tree: the memory hierarchy turned into a data structure
• Athanassoulis et al., “Designing Access Methods: The RUM Conjecture” (EDBT 2016) — short, foundational framing paper. → chapter: reading-rum-conjecture.md — The RUM conjecture: optimize two, pay with the third
• Hellerstein, Stonebraker, Hamilton — “Architecture of a Database System” (2007) — read §1–2 + §6 now for the systems map; the rest is reference material for later topics. → chapter: reading-architecture-of-a-dbms.md — Architecture of a DBMS: the five-box org chart

7. Experiment (in experiments/)

engine_shootout — fjall (LSM) vs redb (B-tree) on the same box, same data, db_bench workload vocabulary (topic 0):

1. fillrandom — N random-key inserts, measure sustained insert throughput.
2. fillseq — same N, sequential keys (B-tree best case: no random page dirtying).
3. readrandom — Zipfian point reads (s=0.99, the M0 generator’s skew) on the loaded DB.
4. scan — full-range iteration throughput.
5. Measure on-disk size after each fill (space amplification, directly).

Rules from topic 0: criterion for the timed loops, report medians, fsync settings identical across engines (fair benchmarking §3.2 — durability parity), record engine versions + config in notes.

Predict the winner of each workload in writing (notes.md) before running. Explaining a wrong prediction is the whole point of the topic.

8. Capstone milestone M1 (in ../../capstone/)

Define the storage-backend abstraction for falkordb-scratch:

• Design the trait first, no peeking at the reference: what operations does a graph engine need from storage? (point get/put, prefix/range scan, atomic batch, snapshot?) Write it down with rationale in capstone/notes/m1-backend-design.md.
• storage crate: trait + in-memory backend (BTreeMap-based is fine — it’s the semantics contract, not the fast path).
• Wire the M0 workload generator through the trait; criterion smoke bench.
• Then read the reference graph/src/storage/backend.rs and write a comparison: what did they need that you didn’t predict, and why?

Done when

• Both experiments’ results are in notes.md with amplification-based explanations, wrong predictions called out explicitly, M1 checklist complete, and you can sketch §1’s two diagrams from memory.

Architecture of a DBMS: the five-box org chart

A database is five cooperating managers, and a storage engine is just one of them. This chapter maps Hellerstein, Stonebraker & Hamilton’s survey — the curriculum’s atlas — onto the topics ahead: read the map chapters this week, then return per-topic as each box gets built. You are NOT reading all ~120 pages now; budget 2 h.

Read NOW (topic 1)

• §1 (main components) — the five-box diagram of a DBMS. Memorize it; it’s the table of contents for topics 3–16:

flowchart TB
    CM["Client communications manager<br/>(topic 7: protocol, RESP)"] --> PC["Process manager<br/>(topic 7/9: threads, admission)"]
    PC --> RP["Relational query processor<br/>parse → rewrite → optimize → execute<br/>(topics 10-11)"]
    RP --> TS["Transactional storage manager<br/>access methods + buffer + locks + log<br/>(topics 1-6, 8-9)"]
    TS --> SC["Shared components<br/>catalog, memory allocator, replication<br/>(topics 15, 22)"]

• §2 (process models) — process-per-worker vs thread-per-worker vs event/async; where admission control lives. Directly informs the capstone server (M7/M9).
• §6 (storage management) — spatial control (why DBs fight the filesystem), buffer pools vs OS page cache, the double-buffering problem. This is the section that justifies this topic’s existence.

Skim NOW, return LATER

Questions to answer in notes.md

1. §6 argues the DBMS should bypass OS caching (O_DIRECT). What are the two distinct problems with letting the OS cache pages? (Double buffering; the OS evicts/flushes with zero knowledge of WAL ordering.)
2. Which of the five §1 boxes does fjall implement? redb? (Neither has a query processor or client manager — “storage engine” ≠ “database”. The capstone builds the other boxes on top, milestone by milestone.)
3. 2007 blind spots: name three things the paper couldn’t see coming. (Candidates: NVMe erasing the seek-time mental model, cloud disaggregation — topic 28, columnar dominance for analytics — topic 12, LSM taking over write paths.)

The one-line takeaway

A database is five cooperating managers, and a storage engine is just one of them — this paper is the org chart for everything the capstone will build.

References

Papers

• Hellerstein, Stonebraker, Hamilton — “Architecture of a Database System” (Foundations and Trends in Databases, 2007) — PDF — read §1–2 + §6 now (2 h); §3–§5 and §7 are reference material to return to per the table above

The B-tree: the memory hierarchy turned into a data structure

Node size = transfer unit, fanout = whatever fits, height = the IO budget — that’s the whole design, and Comer’s 1979 survey is still the cleanest exposition of it in print. This chapter reads it as the theory half of the topic’s B-tree thread: everything in turso’s btree.rs is a footnote to this paper, and §3’s B+ variant is the shape every real engine actually shipped.

Read in this order

1. §1–2 (the problem + the structure) — why balanced trees on disk need high fanout: tree height = number of IOs, and height = log_fanout(n). A 4KB page holding ~100 keys ⇒ 1 billion rows in height 5, of which 3–4 levels cache-resident. This is the whole game.
2. §2.1–2.2 (insertion/deletion) — split on overflow, merge/borrow on underflow. Map to turso: balance_non_root (btree.rs:2995) is the “borrow from siblings first” refinement — Comer calls redistribution out as reducing splits.
3. §3 (B+-tree, B-tree variants)* — the section that matters most:

B-tree:  keys+values in ALL nodes          B+tree: values ONLY in leaves
         ┌─────k,v─────┐                          ┌──────k──────┐  routing only
      ┌─k,v─┐       ┌─k,v─┐                    ┌──k──┐       ┌──k──┐
      ...                                     [k,v|k,v] ↔ [k,v|k,v]  linked leaves
                                                     └── range scan = list walk

Why every real engine chose B+: (a) interior nodes hold only keys → higher fanout → shorter tree; (b) leaf-level linked list → range scans without re-descending; (c) uniform “all data at leaf depth” simplifies everything. 4. §4 (applications: VSAM, etc.) — skim for flavor; 1979’s product landscape.

The paper’s core loop, in the B+ shape §3 argues for — note that the cost of this function is exactly its iteration count:

#![allow(unused)]
fn main() {
// height = number of page reads = ceil(log_fanout(n)) — the whole game
fn lookup(pager: &Pager, root: PageId, key: u64) -> Option<Value> {
    let mut page = pager.read(root);                 // each read: 1 potential IO
    loop {
        match page.kind() {
            Interior => {
                let i = page.keys().partition_point(|&k| k <= key);
                page = pager.read(page.child(i));    // descend one level
            }
            Leaf => return page.find(key),           // B+: values ONLY here;
        }                                            // leaf link → range scans
    }
}
// 4 KB page ≈ 100 keys ⇒ 1 billion rows at height 5, top 3–4 levels cached
}

Questions to answer in notes.md

1. Why do B-trees guarantee ≥50% page occupancy, and what’s the measured average (~69%, ln 2)? Connect to space amplification in the README.
2. B*-tree defers splits by redistributing into siblings. What does turso implement — B+, B*, or a hybrid?
3. Comer’s B-trees assume one page write is atomic. It isn’t (torn writes). Which later machinery patches this hole? (WAL — topic 5; checksums — topic 3.)

The one-line takeaway

The B-tree is the memory hierarchy turned into a data structure: node size = transfer unit, fanout = whatever fits, height = the IO budget.

References

Papers

• Comer — “The Ubiquitous B-Tree” (ACM Computing Surveys 1979) — ~15 pages, 2 h; read §1–3 in order, §3 (the B+/B* variants) matters most, skim §4

Code

• turso core/storage/btree.rs — the living counterpart; walked in reading-turso-btree.md

fjall: the LSM lifecycle in clean Rust

The LSM protagonist of this topic — a codebase small enough that insert-to-SST is traceable in an afternoon, and layered well enough to steal from. fjall is the keyspace/journal/scheduling layer; the actual tree (memtable, SSTs, blooms, block index) lives in the external lsm-tree crate (Cargo.toml:29). Reading fjall shows you the LSM lifecycle; topic 4 descends into lsm-tree itself.

Layout

src/
 ├─ lib.rs               module map — start here
 ├─ keyspace/mod.rs      insert/get/memtable rotation — the heart
 ├─ journal/writer.rs    WAL writes
 ├─ flush/worker.rs      sealed memtable → SST
 ├─ compaction/worker.rs compaction runs
 ├─ supervisor.rs        background orchestration
 ├─ worker_pool.rs       flume-channel thread pool
 └─ poison_dart.rs       panic guard

1. The write path

Start at Keyspace::insert() — src/keyspace/mod.rs:905. Read the whole function; it is the LSM write path diagram from the README:

flowchart LR
    I["insert()<br/>mod.rs:905"] --> J["journal write_raw<br/>mod.rs:928"]
    J --> P["journal persist/fsync<br/>mod.rs:932"]
    P --> M["tree.insert → memtable<br/>mod.rs:940"]
    M --> C["check_memtable_rotate<br/>mod.rs:831"]
    C -- over limit --> R["request_rotation<br/>mod.rs:818"]
    R --> S["inner_rotate_memtable<br/>mod.rs:727<br/>seal + enqueue flush"]
    S --> F["flush::run<br/>flush/worker.rs:12<br/>memtable → SST"]

De-sugared, the function is the topic’s write-path diagram in ten lines:

#![allow(unused)]
fn main() {
fn insert(&self, key: &[u8], value: &[u8]) -> Result<()> {
    let journal = self.journal.lock();          // journal lock BEFORE memtable —
    journal.write_raw(key, value)?;             //   replay order must equal apply order
    journal.persist(self.durability)?;          // fsync per policy, not per write
    let bytes = self.tree.insert(key, value);   // memtable: sorted, in RAM
    self.write_buffer.fetch_add(bytes);         // atomic accounting → backpressure
    if self.memtable_over_size_limit() {
        self.rotate_memtable();                 // seal it + enqueue flush task —
    }                                           //   event-driven, no polling
    Ok(())
}
}

Questions to answer while reading:

• The journal lock is taken before the memtable insert. What ordering bug would reordering them create? (Hint: replay after crash.)
• mod.rs:946 — write buffer accounting is an atomic counter. Where does backpressure actually happen when writers outrun flushing?
• What durability do you get per insert by default — fsync every write, or batched? Compare with what you’ll set in the experiment (durability parity!).

2. The read path

Keyspace::get() — src/keyspace/mod.rs:623 — is two lines: it delegates to tree.get(key, SeqNo::MAX). The run-checking (memtable → sealed → L0… blooms, block index) is all inside lsm-tree. Note where bloom policy is configured: src/keyspace/config/filter.rs:8–43 (BitsPerKey vs FalsePositiveRate, per-level policies — Monkey’s idea productized; topic 4).

3. Compaction scheduling

• Strategies re-exported at src/compaction/mod.rs:7: Leveled, Fifo.
• Worker: compaction/worker.rs:10 — thin: tree.compact(strategy, gc_watermark).
• Trigger plumbing: worker_pool.rs:141–145 sends WorkerMessage::Compact.

The interesting part is what fjall doesn’t do: no compaction geometry here — it delegates policy to lsm-tree, keeping fjall pure lifecycle/scheduling. Good layering to steal for the capstone’s storage crate.

4. Aha spots

1. poison_dart.rs:27–33 — a Drop guard that poisons the whole keyspace if a background worker panics. Crash-visibly instead of serving from corrupt state.
2. ingestion.rs:37–51 — comment explains holding the journal lock across finish() to prevent seqno inversion between writes and bulk ingest. Sequence numbers are the spine of LSM correctness (MVCC preview, topic 8).
3. snapshot_tracker.rs — open-snapshot seqno watermark gates GC: compaction can’t drop a version some reader might still see. This exact problem returns in MVCC vacuuming (topic 8).
4. keyspace/mod.rs:746–750 — rotation immediately enqueues the flush task; no polling anywhere. Event-driven background work via channels.

Done when

You can narrate insert-to-SST without looking, and you know which decisions live in fjall vs lsm-tree.

References

Code

• fjall — src/keyspace/mod.rs (write/read paths), src/journal/writer.rs, src/flush/worker.rs, src/compaction/worker.rs (shallow clone at ~/repos/fjall; line numbers from the clone — expect drift)
• the external lsm-tree crate holds the actual tree (memtable, SSTs, blooms, block index) — topic 4’s territory

The LSM-tree: an IO scheduling policy, not a data structure

Where the origin of the LSM half of the topic’s dichotomy gets read on its own terms. Warning up front: 1996 LSM ≠ 2026 LSM. The paper’s C0/C1 components are B-trees merged by “rolling merge”; modern LSMs (LevelDB lineage) use immutable sorted files + whole-file compaction. Read it for the cost model — that part is timeless — and translate the mechanism as you go.

Why it was written

The motivating workload is TPC-A account history: massive insert rate, few reads. Indexing it with a B-tree means one random page IO per insert. The paper’s thesis: batch inserts in memory, migrate to disk sequentially, and the per-insert IO cost drops by orders of magnitude.

Read in this order

1. §1 (intro + The Five Minute Rule) — the economic argument: pages hot enough are worth keeping in RAM; LSM works because recent data is hot by construction.
2. §2 (two-component LSM) — the core picture. Translate as you read:

paper (1996)                         modern (LevelDB lineage)
─────────────                        ────────────────────────
C0 in-memory AVL/2-3 tree      →     memtable (skiplist)
C1 on-disk B-tree              →     a level of immutable SSTs
rolling merge cursor           →     compaction job
filling disk pages ~100% full  →     SST blocks, sequentially written

The whole 1996 idea fits in one loop — defer, batch, write sequentially, and pay for it at read time:

#![allow(unused)]
fn main() {
fn insert(&mut self, k: Key, v: Val) {
    self.wal.append(&k, &v);          // durability: a sequential append
    self.c0.insert(k, v);             // C0: sorted tree in RAM (≈ memtable)
    if self.c0.bytes() > THRESHOLD {
        // rolling merge: drain C0 into C1 in key order — pages written
        // sequentially, ~100% full; ONE batch amortizes thousands of inserts
        merge_into(&mut self.c0, &mut self.c1);
    }
}

fn get(&self, k: &Key) -> Option<Val> {
    self.c0.get(k).or_else(|| self.c1.get(k))   // the read-amp tax: check
}                                               // EVERY component, newest first
}

3. §3 (cost model) — the payoff. The key result, in modern words: with batching, each insert’s amortized IO cost is ~(entry_size / page_size) × WA sequential bytes instead of one random page read+write. The COST_π algebra formalizes “sequential bandwidth is ~100x cheaper than random IOPS” — the topic 0 ladder in 1996 dollars.
4. §4–5 (multi-component + concurrency/recovery) — skim. Multi-component C0…Ck with size ratio r between adjacent components is exactly modern leveled compaction’s fanout-10 geometry; the optimal-r derivation prefigures Monkey/Dostoevsky (topic 4).
5. §6 (comparison) — skim; the competitors (MD/1 hashing, TSB-tree) are dead, the framing (amortized cost per insert) survived.

Questions to answer in notes.md

1. The paper claims LSM trades what for its insert speedup? (It’s read amp — find where the paper admits point reads must check every component.)
2. Rolling merge keeps C1 a valid B-tree at all times. What do modern LSMs give up by using immutable files instead, and what do they gain? (Hint: crash recovery complexity vs write pattern.)
3. Derive: at size ratio r between components, an entry is rewritten how many times before reaching the last component? Relate to leveled WA ≈ r × levels.

The one-line takeaway

LSM is not a data structure, it’s an IO scheduling policy: convert random writes into sequential ones by deferring and batching — and pay for it at read time.

References

Papers

• O’Neil, Cheng, Gawlick, O’Neil — “The Log-Structured Merge-Tree (LSM-Tree)” (Acta Informatica 1996) — PDF — read §1–3 in order for the cost model; skim §4–6 and translate the mechanism to modern terms as you go

RocksDB: buy the map before walking the territory

RocksDB is everything fjall and tidesdb do, ~50x larger — too big to read, too important to skip. This chapter is not a walkthrough but an orientation map: 30 minutes of ls and header-skimming now, so that when topic 4 (compaction), topic 6 (block cache), and topic 22 (db_bench) ask “where does X live?”, you already know which directory holds the answer.

Directory map

flowchart TB
    API["include/rocksdb/db.h<br/>public API"] --> DBI["db/db_impl/db_impl.h<br/>DBImpl — ~3.8K-line god class"]
    DBI --> MEM["memtable/<br/>skiplist & friends"]
    DBI --> TAB["table/<br/>SST formats<br/>block_based/*"]
    DBI --> VS["db/version_set.h<br/>manifest: which SSTs exist"]
    DBI --> CMP["db/compaction/<br/>compaction_job.h"]
    TAB --> CACHE["cache/<br/>lru_cache.h — block cache"]
    DBI --> FILE["file/ + env/<br/>IO + OS abstraction"]
    DBI --> MON["monitoring/<br/>statistics, histograms"]

The two entry points

• DBImpl::Write() — db/db_impl/db_impl.h:256 (write path entry)
• DBImpl::Get() — db/db_impl/db_impl.h:271 (read path entry)

Everything you traced in fjall/tidesdb exists here too, ~50x larger: journal ↔ db/log_writer.cc, keyspace ↔ column family, manifest ↔ version_set.

Why orient now

When topic 4 asks “how does leveled compaction pick files?”, you should already know the answer lives in db/compaction/ and version metadata in db/version_set.h — navigation cost paid once, here.

References

Code

• rocksdb (shallow clone @ 7c80a5a at ~/repos/rocksdb) — don’t read it yet; orient with the directory map above. Anchors: db/db_impl/db_impl.h, db/version_set.h, db/compaction/, table/block_based/, memtable/skiplist.h, cache/lru_cache.h

The RUM conjecture: optimize two, pay with the third

After the B-tree and LSM papers give the triangle its concrete corners, this short vision paper names the trade-off every storage structure lives inside: read, update, and memory overhead cannot all approach optimal at once. It doesn’t build anything — it hands you the design compass the rest of the curriculum steers by. Read it after the two engine papers.

The claim

For any access method, define overheads relative to the bare minimum work:

• RO (read): bytes read ÷ bytes strictly needed to answer.
• UO (update): bytes written ÷ bytes logically changed.
• MO (memory/space): bytes stored ÷ bytes of live data.

Conjecture: you can optimize any two; the third has a hard lower bound that grows as the other two approach 1. Not a proven theorem — a design compass (hence “conjecture”; the paper is explicit about this).

Read in this order

1. §1–2 — definitions above; make sure you can compute RO/UO/MO for a plain sorted array (RO≈1, UO≈n/2 shifts, MO≈1) and a log (UO≈1, RO≈n, MO grows).
2. §3 (the map) — the paper places real structures on the triangle. Reproduce it:

                       RO = 1 (read-optimal)
                            ▲
                  B+tree ●  │  ● hash index
                            │
             LSM leveled ●  │  ● sorted array (static)
                            │
        LSM tiered ●        │        ● bitmap/bloom (approximate)
                            │
   log ●────────────────────┴────────────────────● compressed archive
 UO = 1 (update-optimal)                   MO = 1 (space-optimal)

3. §4 (moving on the map) — the punchline for this curriculum: knobs are positions, not settings. Bloom bits/key trades MO for RO. Compaction eagerness trades UO for RO. Page fill factor trades MO for UO. Monkey (topic 4) turns this into an actual optimization problem.
4. §5 (research directions) — skim; grade its 2016 predictions with 2026 hindsight (adaptive/learned indexes, versioned data — how did they age?).

Questions to answer in notes.md

1. Place your engine_shootout results on the triangle: which measured number is RO, UO, MO for fjall and redb?
2. Where does FalkorDB’s matrix adjacency sit? (Dense-ish matrix: MO poor for sparse graphs — that’s why delta matrices + roaring exist, topics 20/26.)
3. What’s the RUM position of a WAL by itself? Why does every engine carry one anyway? (Durability isn’t in the triangle — it’s an orthogonal axis the paper deliberately excludes.)

The one-line takeaway

There is no best index, only a workload-shaped position on a three-way frontier — “which engine is better” is an ill-posed question until the workload is named.

References

Papers

• Athanassoulis, Kester, Maas, Stoica, Idreos, Ailamaki, Callaghan — “Designing Access Methods: The RUM Conjecture” (EDBT 2016) — PDF — ~6 pages, 1 h; read after the B-tree and LSM papers so the triangle has concrete corners

tidesdb: the same LSM with nothing abstracted away

The value of this skim (1–2 h) is seeing the machinery you just traced in fjall rendered in plain C, with nothing hidden — memory ordering, pointer arithmetic, and disk offsets are all in your face. Read it as a contrast exercise: match each fjall concept to its C twin and notice exactly what Rust’s abstractions buy you, and what they conceal.

Layout

Write path (file:line)

tidesdb_txn_put            tidesdb.c:26535   stage in per-txn ops array
tidesdb_txn_commit         tidesdb.c:29780   serialize WAL batch → block_manager_write_raw
apply_ops_to_memtable      tidesdb.c:29837   skip-list inserts (atomic refcounts)
rotate check (CAS loop)    tidesdb.c:29850   memtable over threshold → rotate
tidesdb_flush_memtable     tidesdb.c:24887   worker serializes skip list → compressed SST

Read path (file:line)

txn write-set check        tidesdb.c:26672   your own uncommitted writes first
active memtable            tidesdb.c:26808   skip_list_get_with_seq_ref
immutable memtables        tidesdb.c:26845   newest-first, refcount-protected
tidesdb_sstable_get        tidesdb.c:9756    per level: bloom (9810) → block index
                                             binary search (9832) → scan blocks

Exactly the README §1 LSM read diagram, one function per box. Which is to say, in code:

#![allow(unused)]
fn main() {
fn get(&self, key: &[u8]) -> Option<Val> {
    if let Some(v) = self.txn_write_set.get(key) { return Some(v); } // own writes first
    if let Some(v) = self.active_memtable.get(key) { return Some(v); }
    for mt in self.immutable_memtables.newest_first() {              // refcount-pinned
        if let Some(v) = mt.get(key) { return Some(v); }
    }
    for level in &self.levels {
        for sst in level.newest_first() {
            if !sst.bloom.might_contain(key) { continue; }  // skips MOST absent-key IO
            let off = sst.block_index.binary_search(key)?;  // a raw file offset —
            if let Some(v) = sst.read_block_at(off).find(key) {  // the disk format IS
                return Some(v);                                  // the data structure
            }
        }
    }
    None    // read amp made concrete: every stop above was a potential miss
}
}

Compaction

• Enqueue after flush when level over capacity: tidesdb.c:19910.
• Dedup queued work via CAS is_compacting flag: tidesdb_enqueue_compaction, tidesdb.c:25366 — geometry computed at dequeue time, not enqueue.
• Worker picks L_i → L_{i+1} by SSTable counts: tidesdb.c:20143.

What the C makes visible

1. Key+value in one malloc (tidesdb.c:26579): op->value = op->key + key_size — layout as pointer arithmetic. Rust equivalent would be a single Box<[u8]> with split indices; here the trick is load-bearing and explicit.
2. Memory ordering spelled out (tidesdb.c:29761): memory_order_acq_rel on the memtable refcount during rotation. Rust’s Arc hides exactly these barriers — topic 9 makes you write them yourself.
3. Block index returns raw file offsets (tidesdb.c:9835): the reader seeks to a byte position from a struct array. No cursor abstraction — the disk format is the data structure.

Done when

You’ve matched each fjall concept (journal, memtable, rotation, bloom, level) to its C twin and noticed the abstractions Rust buys you — and what they hide.

References

Code

• tidesdb — tidesdb.c (~38K lines, the whole engine), skip_list.c, block_manager.c, bloom_filter.c (~600 readable lines), manifest.c (shallow clone at ~/repos/tidesdb; skim-read, 1–2 h)

Turso’s B-tree: the canonical page engine, in Rust

turso re-implements the SQLite file format, so this is a reading of the canonical page-oriented engine — slotted pages, cursor descent, multi-sibling balance, pager + WAL — with Rust types instead of C macros. It is the B-tree protagonist opposite fjall’s LSM. These files are huge and move fast — expect line-number drift, navigate by symbol name. Two files carry the topic:

1. The slotted page — read this first

core/storage/btree.rs:76–124 has an ASCII diagram of the page layout in the source itself. The shape to internalize:

 ┌────────────┬──────────────────────┬────────────┬─────────────────┐
 │ header     │ cell pointer array   │ free space │ cell content    │
 │ 8/12 bytes │ u16 offsets, →grows  │            │ ←grows, actual  │
 │            │ rightward            │            │ records         │
 └────────────┴──────────────────────┴────────────┴─────────────────┘
   two regions grow toward each other; a "full" page = they meet

• Cell parsing: read_btree_cell() — core/storage/sqlite3_ondisk.rs:816.
• Delete fragmentation + fix: defragment_page() — btree.rs:8422; pointer-array maintenance via copy_within in shift_pointers_left() — btree.rs:9067.

This layout is why B-trees have space amplification: the free gap in the middle of every page is the price of in-place insertion.

The insert, mechanically — two regions growing toward each other until they meet:

#![allow(unused)]
fn main() {
fn insert_cell(page: &mut Page, idx: usize, cell: &[u8]) -> Result<(), Full> {
    let ptrs_end = page.header_len() + 2 * (page.ncells + 1); // ptr array grows →
    let content_start = page.content_start - cell.len();      // content grows ←
    if content_start < ptrs_end {
        return Err(Full);                       // regions met: time to balance/split
    }
    page.buf[content_start..content_start + cell.len()].copy_from_slice(cell);
    page.shift_pointers_right(idx);             // open slot idx — keys stay sorted
    page.write_u16(page.ptr_slot(idx), content_start as u16);
    page.ncells += 1;
    page.content_start = content_start;
    Ok(())
}
// delete = remove the u16 pointer, LEAVE the bytes → fragmentation,
// reclaimed only by defragment_page() — cheap deletes, deferred cleanup
}

2. The cursor — how every operation moves

Main types: BTreeCursor (btree.rs:714), CursorContext (btree.rs:539), PinGuard (btree.rs:375 — pins a page in the cache while the cursor points at it).

• Point lookup: seek() — btree.rs:5681. Trace one descent: root → interior cells binary-searched → child page id → pager fetch → leaf.
• Insert: insert() — btree.rs:5779 → insert_into_page() — btree.rs:2568.

flowchart LR
    S["seek(key)<br/>btree.rs:5681"] --> D["descend: binary search cells,<br/>follow child ptr"]
    D --> PG["pager.read_page<br/>pager.rs:3240"]
    PG --> L["leaf: insert_into_page<br/>btree.rs:2568"]
    L -- page overflows --> B["balance<br/>btree.rs:2793"]
    B --> BNR["balance_non_root<br/>btree.rs:2995<br/>redistribute ≤3 siblings"]

3. Balance ≠ naive split

balance_non_root() (btree.rs:2995) rebalances up to three sibling pages at once, redistributing cells — not the textbook “split one node in two”. This is SQLite’s fill-factor trick: fewer, fuller pages ⇒ lower space amplification and shallower trees. Compare with balance_root() (btree.rs:4774) which grows the tree by one level.

4. Pager + WAL — where “in place” becomes crash-safe

• Pager struct: pager.rs:1335. Reads: read_page() — pager.rs:3240 (cache first), read_page_no_cache() — pager.rs:3185.
• Dirty tracking: add_dirty() — pager.rs:3412. Aha: the page is journaled to the WAL before modification — that’s the write-ahead in write-ahead logging, visible in code.
• WAL: WalFile (wal.rs:2593), frames appended in append_frames_vectored() (wal.rs:708), and checkpoint() (wal.rs:3795) copies frames back into the main DB file. So even the in-place family writes out-of-place first, then reconciles — keep this in mind for the README’s “what is authoritative” framing.
• Page cache: page_cache.rs:99 — SIEVE eviction, default 2000 pages. Buffer-pool preview (topic 6).

Questions to answer

1. How many pages does a point lookup touch on a 1M-row table (page 4KB, ~50 cells interior fanout)? Which of those are realistically cached?
2. Why does balance_non_root prefer redistribution over splitting? What does it do to write amplification (3 dirty pages vs 2)?
3. During checkpoint, what blocks writers? (Read checkpoint() far enough to answer.)

Done when

You can draw the slotted page from memory and explain how one insert can dirty 1 page (common), 3 pages (balance), or O(height) pages (root split).

References

Code

• turso — core/storage/btree.rs (~13K lines: cursor, slotted pages, balance), core/storage/pager.rs, core/storage/wal.rs, core/storage/page_cache.rs, core/storage/sqlite3_ondisk.rs (shallow clone at ~/repos/turso; line numbers drift — navigate by symbol name)

Topic 1 — Notes

Numbers from this machine (Apple Silicon, macOS). Record why, not just what.

Predictions (write BEFORE running the shootout)

Per README §7 — predict the winner and the mechanism, then let the data grade you:

Shootout results

(engine versions: fjall 2.x, redb 2.6 — pin exact versions from Cargo.lock here; durability parity: fjall PersistMode::Buffer vs redb Durability::None.)

• First smoke run (cargo run --release 20000): both engines report ~15x “space amplification” — at 20K × 108B (2.2MB logical) the number is fixed overhead (fjall’s preallocated journal, redb’s initial region sizing), not amplification. Lesson from topic 0: measure at a size where the effect dominates the floor. Re-run at n=1M+ for the real number.

Papers

O’Neil ’96 — LSM-Tree

(questions from reading-lsm-paper.md)

Comer ’79 — The Ubiquitous B-Tree

(questions from reading-comer-btree.md)

RUM Conjecture (EDBT ’16)

(questions from reading-rum-conjecture.md — place shootout results on the triangle)

Architecture of a DBMS (2007)

(questions from reading-architecture-of-a-dbms.md)

Code reading

fjall

turso btree/pager

tidesdb

RocksDB layout

M1 — storage backend abstraction

Design rationale lives in capstone/notes/m1-backend-design.md; comparison with the reference graph/src/storage/backend.rs goes there too (only AFTER the design).

Topic 2 — In-Memory Structures: Hash Tables, Skip Lists, Tries

Redis’s dict, RocksDB’s memtable, Rust’s HashMap — the workhorses of every in-memory database. This topic is where topic 0’s cache lessons become design rules: every structure here is a different answer to “how do I avoid DRAM misses?”

Outcomes

By the end you can:

1. Explain open addressing vs chaining in terms of cache lines, not textbook O(1).
2. Describe incremental rehashing (redis) and SIMD group probing (SwissTable) from memory, and say which latency problem each one solves.
3. Explain why LSM memtables use skip lists instead of hash tables or B-trees.
4. Implement a skip list and an incremental-rehash table, and measure yours honestly against hashbrown / crossbeam-skiplist.

1. Hash tables — the two families and their cache stories

chaining (classic, redis dict):        open addressing (SwissTable/hashbrown):

 buckets   entries (malloc'd)           control bytes      slots (inline)
 ┌───┐    ┌──────┐   ┌──────┐          ┌─┬─┬─┬─┬─┬─┬─┬─┐  ┌────┬────┬────┐
 │ ●─┼───►│k,v,●─┼──►│k,v,∅ │          │h│h│E│h│D│h│E│h│  │ kv │ kv │ kv │…
 ├───┤    └──────┘   └──────┘          └─┴─┴─┴─┴─┴─┴─┴─┘  └────┴────┴────┘
 │ ∅ │     each hop = pointer chase      1 SIMD load checks 8-16 slots at once
 ├───┤     = dependent cache miss        (16 × 7-bit tags in one NEON/SSE2 cmp)
 │ ●─┼───► ...
 └───┘

• Chaining (redis dict): simple, stable pointers, tolerates high load factors — but every collision hop is a dependent DRAM miss (topic 0’s pointer-chase lesson).
• Open addressing (SwissTable): entries inline, probe by scanning control bytes — a separate array of 1-byte tags (7-bit hash + empty/deleted markers). One 16-byte SIMD compare filters 16 slots; you only touch the (cache-line-sized) slot data on a tag match. ~87.5% load factor with almost no probe cost.

The latency-spike problem: a plain table doubling at 100M entries stalls one insert for the whole rehash — a giant hiccup in a server’s p99.9. Two industrial fixes:

flowchart LR
    subgraph REDIS["redis: incremental rehash"]
        A["keep TWO tables<br/>ht[0] old, ht[1] new"] --> B["every add/find moves<br/>1 bucket (dictRehash n=1)"]
        B --> C["rehashidx sweeps until<br/>ht[0] empty → swap"]
    end
    subgraph SWISS["hashbrown: just make rehash fast"]
        D["flat memory, no per-entry<br/>malloc → rehash is a<br/>linear cache-friendly sweep"]
    end

Redis amortizes the spike across operations (reads during rehash check both tables); hashbrown accepts the spike but makes it a memcpy-speed sweep. You will measure both strategies in the experiment — the spike is visible as a max-latency outlier.

2. Skip lists — the memtable’s structure

A sorted linked list with probabilistic express lanes: node height ~ geometric(p).

L3 ──────────────────────────────► 42 ─────────────────────────► ∅
L2 ─────────► 17 ─────────────────► 42 ─────────► 71 ──────────► ∅
L1 ─► 8 ────► 17 ────► 29 ────────► 42 ─► 55 ───► 71 ─► 88 ────► ∅
      search 55: descend when next > target — O(log n) expected

Why memtables (RocksDB, tidesdb) use them instead of:

• hash table — no ordered iteration; flushing to a sorted SST needs sorted data.
• B-tree — needs node splits ⇒ complex latching; a skip list insert touches a handful of independent pointers, so it can be made lock-free with CAS per level.
• the killer feature: memtables are insert-only until frozen, then flushed. No deletes ⇒ no unlink logic ⇒ the lock-free variant stays simple (RocksDB’s InlineSkipList supports concurrent writers with plain CAS loops).

Cache reality check: a skip list is still pointer chasing (topic 0 §2) — each level step is a dependent miss. It wins on concurrency + sortedness, not raw lookup speed; your benchmark will show hashbrown beating it by 5-10x on point lookups. That’s fine — different RUM position.

3. Tries / radix trees — when the key IS the index

radix tree (rax), keys "foo", "foobar", "footer":

        [f o o]  ← compressed run (iscompr): one node holds the shared prefix
           │
        (key: "foo")
         ┌─┴──┐
        [b]  [t]
         │    │
       [a r] [e r]   compressed tails

• Depth = key length, not log n; no hashing, no comparisons — branch on bytes.
• Redis’s rax packs child bytes + unaligned pointers into one flexible array — a node is one cache line for small fanouts.
• ART (the paper) adds adaptive node sizes (Node4/16/48/256) so fanout adapts to density — Node16 is probed with SIMD like SwissTable. Used by DuckDB, HyPer for indexes. The graph-adjacent uses: prefix scans, IP routing, inverted-index terms (topic 23).

4. Cache-conscious layout — the recurring trick

Three structures in this topic all use the same move: separate the “filter” data from the payload so probing touches dense, small memory:

This is the topic 0 flamegraph lesson generalized: SipHash was 21% of lookup cost; control-byte designs make the other 79% (memory stalls) smaller too.

5. Code reading (4–6 h)

• redis dict.c — the incremental rehash machine. → chapter: reading-redis-dict.md — redis dict: rehashing 100M keys without stopping the world
• redis t_zset.c — the skiplist behind sorted sets (spans + rank queries). → chapter: reading-redis-skiplist.md — The redis skiplist: spans make rank queries free
• hashbrown — SwissTable in Rust: control bytes, NEON group probing. → chapter: reading-hashbrown.md — hashbrown: the probe loop the flamegraph couldn’t show
• RocksDB memtable/inlineskiplist.h — lock-free concurrent skiplist. → chapter: reading-rocksdb-memtable.md — InlineSkipList: lock-free by refusing to delete
• redis rax.c — compressed radix tree (skim). → chapter: reading-redis-rax.md — rax: a radix tree packed into cache lines

6. Papers / talks (3–4 h)

• Leis et al., “The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases” (ICDE 2013). → chapter: reading-art-paper.md — ART: sorted like a tree, probed like a hash table
• Matt Kulukundis, “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” (CppCon 2017 — the SwissTable talk). → chapter: reading-swisstable-talk.md — The SwissTable design walk: how benchmarks kill hash tables

7. Experiments (in experiments/)

Implement (this is the topic’s build work — the scaffold compiles with todo!()):

1. src/skiplist.rs — single-threaded skip list (insert, get, ordered iter).
2. src/incremental_map.rs — two-table incremental-rehash hash map, redis-style (migrate ≤ N buckets per operation).

Then bench (benches/structures.rs, harness provided):

• point lookups + inserts vs hashbrown::HashMap, std::BTreeMap, crossbeam_skiplist::SkipMap, sizes 1e3 → 1e7 (Zipfian probes, seed fixed).
• rehash_spike — the headline experiment: insert 10M keys one by one into (a) hashbrown and (b) your incremental map, recording per-insert latency max/p99.9 (HdrHistogram, not criterion — this is a tail-latency question, topic 0 rules). Expect hashbrown to show doubling spikes; yours should flatten them.
• ordered scan — your skiplist vs BTreeMap iteration throughput (the memtable flush path in miniature).

8. Capstone milestone M2 (in ../../capstone/)

• attribute-store crate: string pool (interning: str → u32 id, id → str) + attribute store keyed by (entity id, attr id) + node/edge ID datablocks.
• Design first, then compare with the reference’s attribute_store.rs / string_pool.rs — same no-peeking rule as M1; write the comparison in notes.
• Hash policy decision recorded: default SipHash vs FxHash/ahash for internal maps — justify with topic 0’s flamegraph finding (21% SipHash) + a bench.
• Wire into the workload generator; criterion smoke bench.

Done when

Both structures pass their tests, the rehash-spike plot exists (max latency, incremental vs doubling), benchmark results are explained in notes.md in cache/RUM terms, and M2 is checked off with the reference comparison written.

ART: sorted like a tree, probed like a hash table

The index inside HyPer and DuckDB — a radix tree tuned until it beats hash tables on some workloads while staying sorted. Where rax spends its design budget on memory, ART spends it on lookup speed: node layouts that adapt to fanout, each picking the cheapest search its density allows. It is also where this topic’s SwissTable and radix-tree threads literally meet, in Node16’s SIMD probe.

The problem it solves

Plain radix trees waste memory: a 256-pointer node with 3 children is 2KB of nulls. Binary-comparison trees (B-tree, T-tree) waste time: every level is a key comparison + dependent cache miss. ART’s move: make node size adapt to fanout, so space ≈ compact and depth ≈ radix.

The four node types (§III.A — the core of the paper)

Node4        keys[4]   ┌k┬k┬k┬k┐          linear scan, fits in
             ptrs[4]   └●┴●┴●┴●┘          one cache line

Node16       keys[16]  ┌k×16────────┐     SIMD compare — literally the
             ptrs[16]  └●×16────────┘     SwissTable group probe trick

Node48       index[256]┌256 × 1-byte ─┐   byte-indexed indirection:
             ptrs[48]  └48 × 8-byte  ─┘   index[c] → slot in ptrs

Node256      ptrs[256] ┌●×256────────┐    direct array — no search at all

Nodes grow/shrink between types as children are added/removed. Note the progression of search strategy: linear → SIMD → indexed → direct. Each type picks the cheapest search its density allows.

One match carries the whole idea:

#![allow(unused)]
fn main() {
fn find_child(node: &Node, byte: u8) -> Option<&Node> {
    match node {
        Node4 { keys, ptrs, n } =>              // ≤4 children: linear scan,
            (0..*n).find(|&i| keys[i] == byte)  //   one cache line
                   .map(|i| &ptrs[i]),
        Node16 { keys, ptrs, .. } => {
            let hits = simd_eq(keys, byte);     // the SwissTable group probe
            one_bit(hits).map(|i| &ptrs[i])     //   (≤1 hit here: keys unique)
        }
        Node48 { index, ptrs } =>               // byte-indexed indirection
            slot(index[byte as usize]).map(|s| &ptrs[s]),
        Node256 { ptrs } =>                     // direct — no search at all
            ptrs[byte as usize].as_ref(),
    }
}
}

Reading order

1. §III.A–B — node types + lazy expansion / path compression. Map both onto rax: lazy expansion ≈ rax storing the key tail in a compressed node; path compression ≈ iscompr. ART’s per-node prefix is capped (8 bytes, “pessimistic” overflow re-checks the full key) — rax’s is unbounded. Why does ART cap it? (Fixed-size headers ⇒ no variable-size node layouts.)
2. §III.C–D — insert/delete with node-type transitions. Skim.
3. §III.E + §IV — binary-comparable keys. Don’t skip this. To make ints, floats, strings radix-able you transform them so bytewise order = logical order (flip sign bit, big-endian, etc.). This idea is everywhere: RocksDB comparators, FoundationDB tuples, your capstone’s composite (entity,attr) keys in M2.
4. §V — evaluation. Read Fig. 8/9 with topic-0 eyes: where does ART beat the hash table (dense integer keys — short paths, no hash cost) and where does it lose (long random strings — depth ∝ length)?

Space guarantee worth remembering

§III.B proves worst-case 52 bytes per key regardless of key distribution — the adaptive nodes + path compression make the bound possible. Compare: your skiplist’s per-node cost (1.33 pointers avg + key) has no such bound story.

Questions to answer in notes.md

1. Node16 search is the SwissTable group probe (compare 16 bytes in one SIMD op). What’s the structural difference between how ART and SwissTable use the result? (ART: index into child pointers; Swiss: candidate slots to verify.)
2. Height of ART on 8-byte integer keys is ≤ 8 regardless of n. At what n does log₂(n) exceed that — i.e., where does a B-tree start losing on depth alone?
3. For the capstone: would ART beat your M2 hash-based attribute store for (entity id, attr id) → value? Sketch the key encoding and the RUM trade.

Done when

You can name the four node types with their search strategies from memory, and explain binary-comparable key encoding well enough to encode (u64, u16) pairs.

References

Papers

• Leis, Kemper, Neumann — “The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases” (ICDE 2013) — PDF — ~2 h; §III.A is the core, don’t skip §III.E/§IV (binary-comparable keys), read §V’s figures with topic-0 eyes

hashbrown: the probe loop the flamegraph couldn’t show

This IS std::collections::HashMap — you profiled it in topic 0 (21% SipHash, rest inlined probe loop), and now you read the probe loop the flamegraph flattened into “everything else”. One idea carries the whole design: keep a dense array of 1-byte tags beside the slots, so one SIMD load filters 8–16 candidates before a single key byte is touched.

1. The control-byte array — the whole idea

Every slot has a 1-byte tag in a separate dense array (src/control/tag.rs:9–49):

tag values:  EMPTY = 0xff   DELETED = 0x80   FULL = 0b0xxxxxxx (h2: top 7 hash bits)

hash (64 bits): ┌──────── h1: index bits ────────┬─ h2: top 7 ─┐
                └── which group to probe first ──┴─ tag value ─┘

control array:  [23|EMPTY|91|07|DELETED|55|23|EMPTY| ... ]
                 └────────── one 8/16-byte SIMD load ─────────┘
slot array:     [ kv | ___ | kv | kv | ___ | kv | kv | ___ ]  touched only on tag hit

Probing = compare h2 against 16 tags in one SIMD op; only matching slots get a real key comparison. False-positive rate per group ≈ 16/128 — cheap. This is the “dense filter + fat payload” pattern (README §4).

The lookup, de-macro’d:

#![allow(unused)]
fn main() {
fn find(table: &RawTable, hash: u64, key: &K) -> Option<usize> {
    let h2 = (hash >> 57) as u8;                        // top 7 bits = the tag
    let mut probe = ProbeSeq::new(h1(hash), table.mask); // triangular stride
    loop {
        let group = Group::load(&table.ctrl[probe.pos]); // ONE dense cache line
        for bit in group.match_tag(h2) {                 // SIMD: 8–16 tags at once
            let slot = (probe.pos + bit) & table.mask;
            if table.key(slot) == key { return Some(slot); } // 2nd line: the slot
        }
        if group.match_empty().any_bit_set() {
            return None;    // EMPTY stops the probe; DELETED does NOT —
        }                   //   the key may have been pushed past a tombstone
        probe.move_next(table.mask);
    }
}
}

2. Where things live

3. Read in this order

1. tag.rs — EMPTY/DELETED encoding. Why is EMPTY 0xff and full tags 0b0xxxxxxx? (So match_empty_or_deleted = “high bit set” — one SIMD sign test.)
2. group/neon.rs:78–90 — the 8-byte NEON group ops (Apple Silicon path). Note x86 SSE2 gets 16-wide groups; ARM gets 8. Measurable? (Experiment idea.)
3. raw.rs:76–93 — ProbeSeq: stride grows by one group per step (triangular numbers). The comment links the proof that mod-power-of-two triangular probing visits every group exactly once — no cycling, no missed slots.
4. Insert path raw.rs:1952 — find first EMPTY or DELETED. Tombstone subtlety (raw.rs:1033–1043): inserting over DELETED doesn’t consume growth_left, and a table full of tombstones triggers rehash-in-place instead of growth.
5. Aha: the trailing mirror — raw.rs:223: the control array allocates buckets + Group::WIDTH bytes; the tail replicates the head so a group load starting near the end never wraps. Branchless boundary handling paid in 16 bytes.

4. Connect to your topic 0 numbers

Your flamegraph showed the probe loop fully inlined and memory-stall-bound at 10M keys. Now you can name what’s stalling: the control-byte load is the one guaranteed miss per probe (dense array, ~1 cache line per group); the slot touch is the second. h2 filtering exists precisely so there’s rarely a third.

Questions to answer in notes.md

1. Why 7/8 load factor rather than redis’s 1.0? (Open addressing degrades near full — probe lengths explode; chaining just grows chains linearly.)
2. Rust 2018 chose SipHash default for HashMap (DoS resistance) — after this reading plus the 21% flamegraph number, write the one-paragraph policy for the capstone: where FxHash/ahash, where SipHash stays.
3. What does DELETED do to a long-lived table with churn? Relate to LSM tombstones — same problem, same fix (rewrite/compact).

Done when

You can draw the control-byte array and narrate one lookup from hash to slot, including both cache lines it touches.

References

Code

• hashbrown (shallow clone at ~/repos/hashbrown) — src/raw.rs (RawTable, ProbeSeq, insert path), src/control/tag.rs, src/control/group/neon.rs (the Apple Silicon path; SSE2 sibling for x86)

redis dict: rehashing 100M keys without stopping the world

A hash table serving 100K ops/s cannot stop the world to rehash 100M entries — the resulting p99.9 spike would be a service outage. This chapter walks redis’s answer, the topic’s first industrial latency fix: keep two tables and migrate one bucket at a time, piggybacked on normal operations. It is also the design you’ll replicate in this topic’s experiment.

1. The two-table struct

dict.h:143–159 — the whole design in one struct:

struct dict {
    dictType *type;
    void **ht_table[2];        // ht[0] = old, ht[1] = new (during rehash)
    unsigned long ht_used[2];
    long rehashidx;            // -1 = not rehashing; else next bucket to migrate
    int16_t pauserehash;
    signed char ht_size_exp[2]; // sizes as exponents: size = 1 << exp
};

flowchart LR
    OP["any dictAdd/dictFind<br/>dict.c:635 / dict.c:779"] --> STEP["_dictRehashStepIfNeeded<br/>dict.c:1705"]
    STEP --> RH["dictRehash(d, 1)<br/>dict.c:405 — move 1 bucket<br/>ht[0]→ht[1]"]
    RH --> DONE{"ht[0] empty?"}
    DONE -- yes --> SWAP["free ht[0], ht[1]→ht[0]<br/>rehashidx = -1"]
    DONE -- no --> OP

2. dictRehash — dict.c:405

Read the whole function (~50 lines):

• empty_visits = n*10 (dict.c:406) — the cap on empty buckets visited per step. Question: why is this needed? (A sparse old table would otherwise make one “step” scan unboundedly far — the amortization guarantee would silently break.)
• Each migrated bucket’s chain is walked and every entry re-hashed into ht[1] (dict.c:420–431). Note: entries move one bucket at a time, not one entry.

The whole machine, distilled:

#![allow(unused)]
fn main() {
fn rehash_step(d: &mut Dict, mut buckets: usize) {
    let mut empty_visits = buckets * 10;         // cap the sparse-table scan
    while buckets > 0 && d.used[0] > 0 {
        while d.ht[0].bucket(d.rehashidx).is_empty() {
            d.rehashidx += 1;
            empty_visits -= 1;
            if empty_visits == 0 { return; }     // bounded work per op — the point
        }
        for entry in d.ht[0].take_bucket(d.rehashidx) {
            let idx = entry.hash & d.mask[1];    // re-hash into the NEW table only
            d.ht[1].push_bucket(idx, entry);
        }
        d.rehashidx += 1;
        buckets -= 1;
    }
    if d.used[0] == 0 { d.swap_tables(); d.rehashidx = -1; }
}
// every dictAdd/dictFind calls rehash_step(d, 1) — and during the migration,
// every lookup must check BOTH tables
}

3. Who pays the rehash tax

• dictAddRaw — dict.c:635; dictFind — dict.c:779; dictAddOrFind — dict.c:1742. Every read and write does one step. During rehash, lookups must check both tables (new keys go only to ht[1]; the key you want may be in either).
• Cost model: rehash O(n) total, amortized O(1) per op, worst per-op ≈ one bucket chain + 10 empty visits. This is the design you’ll replicate in the experiment.

4. Resize policy — dict.c:1638

• Grow at load factor 1.0 (ht_used >= size) when resizing is enabled; forced grow at dict_force_resize_ratio even when disabled (dict.c:1655 — resizing gets disabled during fork/BGSAVE to avoid COW page storms — a durability-meets-data-structure interaction worth pausing on).

5. dictScan — the reverse-binary trick (dict.c:1518)

How do you iterate a table that may rehash under you without missing or endlessly duplicating keys? dictScan increments the cursor in reversed bit order (dict.c:1579–1615). Read the long comment above it — one of the great comments in open source. The property: buckets already visited at size 2^n map onto already-visited buckets at size 2^(n+1). Guarantee: every key present for the whole scan is returned ≥ once (duplicates possible, misses not).

6. Contrast: valkey’s libvalkey client dict

~/repos/valkey/deps/libvalkey/src/dict.c — a single-table, full-rehash dict (dict.c:103–150): no rehashidx, no two-table dance. Fine for a client’s small maps; unacceptable for a server’s keyspace. Same structure, different RUM position — latency requirements are part of the workload.

Questions to answer in notes.md

1. During rehash, dictAddRaw inserts only into ht[1]. Why is inserting into ht[0] a correctness bug, not just a wasted move?
2. What does pauserehash exist for? (Hint: safe iterators.)
3. Redis caps empty_visits at 10n. What tail-latency guarantee does that give one operation, in buckets touched?

Done when

You can implement the two-table scheme from memory — you’ll do exactly that in this topic’s experiment.

References

Code

• redis src/dict.c, src/dict.h — line numbers from the local clone; the dictScan comment (dict.c:1518) is one of the great comments in open source
• valkey deps/libvalkey/src/dict.c — the single-table, full-rehash contrast case

rax: a radix tree packed into cache lines

Redis’s compressed radix tree — behind stream IDs, client tracking keys, and cluster slot→key maps — is what a trie looks like when memory is the corner of the RUM triangle you’re defending: one variable-size node layout, deliberately unaligned pointers, path-compressed runs. Read for the layout (~45 min, skim the insert logic); it’s the memory-first contrast case for the ART paper that follows.

1. The node — rax.h:78–111

typedef struct raxNode {
    uint32_t iskey:1;     /* this node terminates a key             */
    uint32_t isnull:1;    /* key has no associated value            */
    uint32_t iscompr:1;   /* node is a compressed run               */
    uint32_t size:29;     /* # children (or run length if iscompr)  */
    unsigned char data[]; /* EVERYTHING else lives here             */
} raxNode;

Four bytes of header, then one flexible array holding child bytes, child pointers, and the optional value pointer, all packed:

non-compressed, size=3 ("abc" branches):        compressed run "xyz" (iscompr=1):

┌header┐┌── data[] ─────────────────────┐       ┌header┐┌── data[] ────────────┐
│4 bytes││a b c pad│ A* │ B* │ C* │ V*? │       │4 bytes││x y z pad│ Z* │ V*?  │
└──────┘└─────────┴────┴────┴────┴─────┘       └──────┘└─────────┴────┴──────┘
          ▲ char bytes first (dense filter!)      whole run = ONE child pointer
          then pointers, then value if iskey       (points past the run)

• Layout comment at rax.h:83–109 — read it in full; it’s the spec.
• A compressed node stores a multi-byte run (“foo”) with a single child pointer — that’s the path compression that keeps depth ≈ distinct branches, not key length.

2. The unaligned-pointer aha — rax.h:90, 99

The child pointers in data[] are not aligned: chars come first, so a pointer may start at any byte offset. Redis reads/writes them with memcpy (raxNodeLastChildPtr, raxNodeFirstChildPtr helpers). Why tolerate that?

• One allocation per node; header + chars + pointers usually fit one cache line for small fanouts.
• Scanning the char bytes to pick a branch touches only the dense prefix of the node — same “dense filter, fat payload” move as SwissTable control bytes (README §4). Alignment padding would spread the node across lines.

Modern ARM/x86 do unaligned loads nearly free; the cache line saved is worth more.

3. Insert = split machinery — rax.c:515–658 (skim)

raxGenericInsert walks with raxLowWalk, which returns splitpos — where the new key diverges inside a compressed run. The walk itself is the tree’s whole read path:

#![allow(unused)]
fn main() {
// returns (bytes of key consumed, split position inside a compressed run)
fn low_walk(mut node: &RaxNode, key: &[u8]) -> (usize, usize) {
    let mut i = 0;
    while i < key.len() {
        if node.iscompr() {
            let run = node.chars();                  // e.g. "oot" — one node
            let m = common_prefix(run, &key[i..]);
            if m < run.len() { return (i + m, m); }  // diverged MID-run: splitpos
            i += m;
            node = node.child(0);                    // whole run = ONE pointer
        } else {
            match node.chars().iter().position(|&c| c == key[i]) { // dense scan:
                Some(j) => { node = node.child(j); i += 1; }       //   chars only,
                None => return (i, 0),                             //   ptrs untouched
            }
        }
    }
    (i, 0)      // consumed the whole key: node.iskey ⇒ hit
}
}

The long comment before the insert code enumerates the cases; the picture:

insert "first" into node ["footer"]:   split the run at splitpos=1
              [f]                ← shared prefix survives as run (or single node)
             ┌─┴─┐
        ["ooter"] ["irst"]       ← two compressed tails, new branching node

Every case is “cut the run, make a 2-child branching node, re-hang the tails”. Don’t memorize the five cases — just verify the invariant: after any insert, no node has exactly one child unless it’s compressed (otherwise it would be merged into a run).

4. Contrast with ART (next reading)

Same structure, opposite RUM corner: rax minimizes M, ART minimizes R.

Questions to answer in notes.md

1. Why does rax put the char bytes before the pointers instead of interleaving (char,ptr) pairs? (Branch decision reads only chars — one dense scan.)
2. A radix tree has no hash function and no key comparisons — what does it give up vs a hash table? (Point-lookup cost ∝ key length; but you gain prefix scans and ordered iteration — which topic 23’s inverted index will want.)

Done when

You can sketch a compressed vs non-compressed node’s data[] layout from memory and say why the pointers are unaligned on purpose.

References

Code

• redis src/rax.h, src/rax.c — the layout comment at rax.h:83–109 is the spec; read it in full before the functions

The redis skiplist: spans make rank queries free

The canonical readable skiplist — the structure behind ZADD/ZRANGE/ZRANK in t_zset.c — with one addition the textbooks skip: every forward link records how many level-0 nodes it jumps over, so summing spans during an ordinary descent yields a node’s rank at no extra cost. Read it before the RocksDB memtable chapter to see what a skiplist looks like when concurrency isn’t allowed to take features away.

1. The structs — server.h:1699–1716

typedef struct zskiplistNode {
    sds ele; double score;
    struct zskiplistNode *backward;      // level-0 doubly-linked
    struct zskiplistLevel {
        struct zskiplistNode *forward;
        unsigned long span;              // # of L0 nodes this link jumps over
    } level[];                           // flexible array: height varies per node
} zskiplistNode;

Two things beyond the textbook skiplist:

• span — each forward link records how many level-0 nodes it skips. Summing spans during a descent = the node’s rank, for free. That’s ZRANK/ZRANGE-by-index in O(log n) without any extra structure.
• backward — level-0 only, making reverse range queries (ZREVRANGE) a plain list walk from the tail.

The span trick in action — an ordinary descent that counts as it goes:

#![allow(unused)]
fn main() {
fn rank_of(list: &SkipList, target: &Key) -> u64 {
    let mut node = &list.head;
    let mut rank = 0u64;
    for lvl in (0..list.level).rev() {           // express lanes: top → bottom
        while let Some(next) = node.forward(lvl) {
            if next.key < *target {
                rank += node.span(lvl);          // spans sum to the rank — free
                node = next;
            } else {
                break;                           // too far: drop one level
            }
        }
    }
    rank        // ZRANK in O(log n), no auxiliary structure, no re-walk
}
}

2. Height selection — t_zset.c:254

zslRandomLevel(): geometric with p = 0.25 (ZSKIPLIST_P, server.h:630), max level 32. Compare: RocksDB uses branching factor 4 (same p) but caps at 12. Question: expected pointers per node at p=0.25? (1/(1−p) = 1.33 — vs 2 for a binary tree.)

3. zslInsert — t_zset.c:265–339

The heart. The descent records, per level:

• update[i] — the rightmost node at level i that precedes the insert point (the nodes whose forward pointers must be spliced);
• rank[i] — cumulative span up to update[i] (so new spans can be computed without re-walking).

insert 55, height 2:                       update[] captured on the way down
L2 ──────► 17 ────────────────► 71        update[2]=17  rank[2]=2
L1 ──────► 17 ────► 42 ─[55]──► 71        update[1]=42  rank[1]=3   splice
L0 ─► 8 ─► 17 ─► 29 ─► 42 ─[55]► 71       update[0]=42  rank[0]=3   splice
                                           levels above height: span += 1 only

Note the span bookkeeping at t_zset.c:304–305: levels above the new node’s height don’t get a new link, but their spans still grow by one — subtle, and the kind of invariant your own implementation will get wrong first try.

4. What redis does NOT do

No locks, no CAS — redis is single-threaded on the data path, so this skiplist is free to use backward pointers and spans (both hard to maintain lock-free). Contrast with RocksDB’s InlineSkipList (concurrent writers ⇒ no backward pointers, no spans, no deletes). Concurrency removes features — a theme topic 9 makes precise.

Questions to answer in notes.md

1. Why does the zset need both the skiplist and a dict (score lookup by member)? What does that cost in memory, and what’s the RUM read?
2. Derive the expected search cost at p=0.25: levels × nodes-per-level ≈ log₄(n) × ~3 compares. At n=1M: ~30 dependent pointer hops — now price it with topic 0’s ladder (30 × ~100ns if cold). Compare your measured number.

Done when

You can explain spans to someone in two sentences, and you know which features your experiment’s skiplist can steal (backward/span) vs what RocksDB’s concurrency forbids.

References

Code

• redis src/t_zset.c (zslInsert, zslRandomLevel) — struct definitions in src/server.h:1699–1716

InlineSkipList: lock-free by refusing to delete

This is where LSM write throughput lives: every Put in half the industry lands in this one header. Two ideas are the whole file — a node layout that puts the hot pointer and the key on the same cache line by indexing the tower negatively, and a concurrency contract kept simple by one workload restriction: memtables never delete, they freeze and drop wholesale. Budget: 1–2 h.

1. The node layout trick — read this first

Lines 358–421. A node is one allocation, three regions, and the struct points at the middle:

 raw allocation (from concurrent arena, line 860-869):

 ┌──────────────────────────┬───────────────┬─────────────────┐
 │ tower: next_[h-1]…next_[1]│ Node: next_[0]│ key bytes inline│
 └──────────────────────────┴───────────────┴─────────────────┘
                              ▲ Node* points HERE
   levels accessed by NEGATIVE indexing: (&next_[0] - n)      line 383
   key accessed as (&next_[1]):                               line 374

Why: the common case touches next_[0] and the key — adjacent, same cache line(s). Taller levels (rare) sit before the node. No separate key allocation, no pointer to the key. This is README §4’s dense-filter/inline-payload pattern again, by an author who priced the cache lines.

2. The concurrency contract

• Next()/SetNext() — acquire/release (lines 383, 390); CASNext — line 395.
• InsertConcurrently — line 913; the CAS loop lines 1135–1171: compute splice (prev/next per level), CAS level 0 first? No — read carefully: which level is linked first, and why does a partially linked node never break readers? (A node visible at level i but not i+1 is just… slower to find. Correctness needs only level 0.)
• No deletes, no unlink (comment lines 31–33): memtables are frozen then dropped wholesale. This one workload restriction is what keeps the lock-free code ~200 lines instead of a research paper (no hazard pointers to unlink, nothing is ever freed while readers run). Constraint-driven simplicity — the design lesson of the whole file.

The concurrent insert, reduced to its CAS skeleton:

#![allow(unused)]
fn main() {
fn insert_concurrently(list: &SkipList, node: &Node, height: usize) {
    let mut splice = list.find_splice(node.key());       // prev/next per level
    for lvl in 0..height {                               // bottom-up: correctness
        loop {                                           //   needs only level 0
            node.set_next(lvl, splice.next[lvl]);        // prepare BEFORE publish
            if splice.prev[lvl]
                .cas_next(lvl, splice.next[lvl], node)   // release: key bytes are
            {                                            //   visible before the link
                break;
            }
            splice.recompute(lvl, node.key());           // lost the race — re-find
        }                                                //   neighbors, retry
    }
}
// a node linked at level 0 but not yet above is merely slower to find —
// never incorrect. That asymmetry is what makes the lock-free version small.
}

3. Supporting cast

• RandomHeight — lines 559–573, branching factor 4, max 12 levels.
• Arena: memory/concurrent_arena.h:57–68 — per-core shards so concurrent inserts don’t contend on the allocator either.
• The plug into the engine: memtable/skiplistrep.cc:17–397 implements MemTableRep; siblings in memtable/: hash_skiplist_rep (hash → per-bucket skiplists, for point-heavy), hash_linklist_rep, vectorrep (bulk-load: append then sort-on-flush). The memtable is pluggable because RUM positions differ per workload — RocksDB ships four answers.

Questions to answer in notes.md

1. Redis’s skiplist has spans + backward pointers; this one has neither. For each, say exactly what breaks under concurrent CAS inserts.
2. Why acquire/release on the links rather than SeqCst? What reorder is actually being prevented at line 383? (Reader must see the node’s key bytes written before the pointer that publishes it — classic publish pattern, topic 9.)
3. Estimate: at branching 4 and 1M entries, how many dependent misses per lookup, and why does your hashbrown number from topic 0 beat it? Where does the skiplist still win? (Sorted iteration for flush; concurrent writers.)

Done when

You can explain the negative-index tower AND why insert-only makes lock-free easy — these two ideas are the file.

References

Code

• rocksdb memtable/inlineskiplist.h — the header comment (lines 31–33) states the no-delete contract; also memory/concurrent_arena.h:57–68 (sharded arena) and memtable/skiplistrep.cc (the MemTableRep plug-in point and its three siblings)

The SwissTable design walk: how benchmarks kill hash tables

How Google replaced std::unordered_map fleet-wide — told as a sequence of designs, each rejected by a measurement. This chapter is a watching guide for Kulukundis’s CppCon talk: watch it after reading reading-hashbrown.md, because the talk is the design narrative for the code you just read. Budget ~60 min video + 30 min notes.

Why watch a talk about a table you already read

The hashbrown source shows the final design. The talk shows the sequence of rejected designs and the benchmark that killed each one — it’s a masterclass in the topic-0 method (hypothesize → measure → iterate).

The design walk (watch for these beats)

std::unordered_map          chaining, per-node malloc, iterator stability
        │  "every lookup = 2+ dependent misses"
        ▼
dense_hash_map              open addressing, quadratic probe, but 2 sentinel
        │                   keys stolen from the user + 50% max load
        ▼
"store metadata per slot"   1 byte: empty/deleted/full + 7 hash bits
        │  "but scanning bytes one at a time is slow"
        ▼
SwissTable                  group the bytes, compare 16 at once with SSE2
                            → 87.5% load factor, ~1 miss per lookup

Timestamps are approximate across uploads — navigate by slide titles instead:

• “The C++ standard basically mandates chaining” — why unordered_map can’t be fixed in place (pointer stability + bucket API promises).
• The metadata byte slide — the h2/control-byte idea introduced.
• The SSE2 _mm_movemask_epi8 slide — the group probe; this is hashbrown’s Group::match_tag, NEON on your machine.
• Load factor + tombstone discussion — where the 7/8 and rehash-in-place decisions come from (hashbrown raw.rs:152, 1033).

Connect to what you’ve read

Questions to answer in notes.md

1. Google couldn’t ship this as std::unordered_map because the standard’s API promises (pointer stability, bucket interface) mandate chaining. Which redis dict features would SwissTable similarly break? (Incremental rehash needs stable entries? Check — redis moves entries between tables anyway; the real conflict is dictScan’s bucket cursor.)
2. The talk reports big fleet-wide RAM savings from the load-factor jump (50% → 87.5%) plus removing per-node mallocs. Estimate the bytes-per-entry difference for a u64→u64 map: chaining with malloc’d nodes vs SwissTable at 7/8 load. Show the arithmetic in notes.
3. Kulukundis says hash quality matters more for open addressing than chaining — why? (Clustering compounds; a bad h2 also raises false positives.)

Done when

You can retell the rejected-design sequence (chaining → dense_hash_map → metadata bytes → SIMD groups) and give the one-line benchmark reason each step was taken.

References

Papers

• Kulukundis — “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” (CppCon 2017 talk) — video — ~60 min; timestamps vary across uploads, navigate by the slide titles listed above

Code

• hashbrown — the Rust incarnation of the final design; walked in reading-hashbrown.md

Topic 2 — notes

Predictions (fill BEFORE running benches)

Reading answers

redis dict (reading-redis-dict.md)

1. Insert into ht[0] during rehash — why a bug:
2. pauserehash exists for:
3. empty_visits=10n tail guarantee:

redis skiplist (reading-redis-skiplist.md)

1. Why skiplist + dict both:
2. Expected search cost at p=0.25, priced vs measured:

hashbrown (reading-hashbrown.md)

1. 7/8 vs 1.0 load factor:
2. Hash policy paragraph (for M2 decision):
3. DELETED churn ↔ LSM tombstones:

RocksDB memtable (reading-rocksdb-memtable.md)

1. spans/backward under concurrent CAS:
2. acquire/release vs SeqCst at line 383:
3. Miss estimate vs hashbrown number:

rax / ART / SwissTable talk

• (questions in each guide)

Experiment findings

• rehash_spike table + per-decile max:
• Where my skiplist loses to hashbrown and by how much (RUM terms):
• Implementation trade I chose for skiplist node layout, and why:

M2 log

• attribute-store design written BEFORE peeking at reference
• comparison vs reference attribute_store.rs / string_pool.rs:
• hash policy decision + bench evidence:

Topic 3 — B-Tree Internals & Paged Storage

Pages are how disks think. SQLite’s btree.c, turso’s Rust re-implementation, and LMDB’s copy-on-write variant are three answers to the same question: how do you keep a sorted map in fixed-size blocks that survive power loss?

Outcomes

By the end you can:

1. Draw a slotted page from memory: header, cell pointer array, cell content area, freeblock chain — and explain why the two regions grow toward each other.
2. Narrate a node split (SQLite’s 3-sibling balance) and say why it’s ≤3.
3. Explain LMDB’s no-WAL commit (COW + double meta page) and its costs.
4. Build a disk B+tree with 4KB slotted pages and bench it honestly vs redb.

1. The slotted page — one picture to rule the topic

 4096-byte page (SQLite/turso format):

 ┌──────────────┬─────────────────┬─────────▼── grows down ──┬──────────────┐
 │ header 8/12B │ cell ptr array  │      free space          │ cell content │
 │ type,frag,   │ [u16,u16,u16..] │                          │ (cells added │
 │ freeblock,   │  sorted by KEY  │   freeblocks chained     │  right→left, │
 │ ncell,cstart │  grows up ▲     │   through here too       │  any order)  │
 └──────────────┴─────────────────┴──────────────────────────┴──────────────┘

 binary search touches ONLY the ptr array (dense) → then one jump to the cell
 delete = remove ptr + add freeblock; insert = find slot (allocateSpace) or defrag

The indirection is the whole trick: cells never move on insert/delete of others (pointers do), so binary search stays cheap and deletion is O(1) + freeblock bookkeeping. This is the dense-filter/fat-payload pattern (topic 2 §4) on disk: the pointer array is the filter.

Header fields (turso btree.rs:76–124, spec in SQLite btreeInt.h:1–215): byte 0 page type; 1–2 first freeblock; 3–4 cell count; 5–6 content-area start; 7 fragmented bytes; 8–11 rightmost child pointer (interior only).

2. The four cell types (SQLite family)

Payload > maxLocal spills to an overflow chain: keep minLocal + (n − minLocal) % (usable − 4) bytes local, rest in a linked list of overflow pages (last 4 local bytes = first overflow page number). The formulas (maxLocal = (usable−12)·64/255 − 23, etc.) look arbitrary — they guarantee each page holds ≥4 cells so the tree keeps fanout even with fat keys.

3. Splits and balancing — where the complexity lives

flowchart TD
    INS["insert overflows page"] --> Q{"rightmost leaf,<br/>append pattern?"}
    Q -- yes --> BQ["balance_quick:<br/>new right sibling,<br/>1 divider up<br/>(btree.c:8039)"]
    Q -- no --> BNR["balance_nonroot:<br/>pool cells of page +<br/>≤2 siblings + dividers,<br/>redistribute evenly<br/>(btree.c:8277)"]
    BNR --> DEEP{"root itself full?"}
    DEEP -- yes --> BD["balance_deeper: new root,<br/>tree grows UP (btree.c:9081)"]

• NB = 3 (btree.c:7552): balance pools at most 3 sibling pages. SQLite’s comment: the right-bias tweak alone made the whole database “about 25% faster” — splits are hot.
• Deletion from an interior node: swap with the predecessor from the leaf level, then rebalance the leaf (btree.c:9873) — interior deletes reduce to leaf deletes.
• Tree grows up (new root), never down — parent pointers stay implicit in the cursor stack.

4. LMDB — the copy-on-write counterpoint

 commit N (writes pages 7',3',root'):          the two meta pages:

     meta0(txn N-2)  meta1(txn N-1)            ┌───────────────────────────┐
          │               │ ◄─ readers          │ commit = write dirty pages │
          ▼               ▼                     │        + fsync             │
        [root]         [root']                  │        + write meta[N%2]   │
        /    \         /    \                   │        + fsync             │
      [3]    [7]     [3']   [7']                │ crash anywhere ⇒ old meta │
              ▲ old pages still valid           │ still valid. NO WAL.      │
                (readers may hold them)         └───────────────────────────┘

• Never overwrite: mdb_page_touch (mdb.c:3015) copies any clean page before the first write in a txn; the whole root-to-leaf path gets new page numbers.
• Commit = flush dirty pages, fsync, write the other meta page, fsync (mdb_env_write_meta, mdb.c:4847, slot txnid & 1). Torn writes can’t corrupt: the previous meta still points at a complete old tree.
• Old pages are recycled through a freelist DB once the oldest reader (mdb_find_oldest, mdb.c:2640) has moved past them — MVCC GC as a data problem.
• Cost: write amp (whole path copied per commit), single writer, and the reader table pins pages (a stuck reader = unbounded growth). Same trade the reference capstone’s cow_btree makes in memory — compare deliberately in M3 notes.

5. Code reading (5–7 h)

• turso core/storage/btree.rs — deep dive: slotted page ops, balance state machines, overflow, freelist. → chapter: reading-turso-btree-deep.md — Inside the slotted page: freeblocks, overflow, balance
• SQLite src/btree.c — the classic (11.6K lines; guided skim). → chapter: reading-sqlite-btree.md — btree.c: twenty years of production scars
• LMDB libraries/liblmdb/mdb.c — COW, double meta, no WAL. → chapter: reading-lmdb.md — LMDB: recovery is choosing a root pointer

6. Papers / docs (3–4 h)

• Graefe, “Modern B-Tree Techniques” (Foundations & Trends 2011) — the survey; read selectively. → chapter: reading-graefe-survey.md — Modern B-tree techniques: height is the metric, fanout is the lever
• SQLite database file format (official doc) — read alongside the code. → chapter: reading-sqlite-file-format.md — The SQLite file format: decode a row by hand

7. Experiments (in experiments/)

Implement a slotted-page disk B+tree, fixed 4KB pages (scaffold compiles with todo!(); page-format helpers + tests provided):

1. src/page.rs — slotted page: header, cell ptr array, insert/delete/defrag.
2. src/btree.rs — B+tree on a page file: search, insert with leaf split, range scan via leaf sibling links.

Then bench (benches/disk_btree.rs):

• point lookups + range scans vs redb, 1M keys, cold-ish (drop your page cache between runs is impractical — note it; compare warm numbers honestly).
• prefix truncation experiment: keys = 32-byte strings sharing 24-byte prefixes. Measure fanout (keys/page) and lookup speed with full keys vs suffix-truncated separators in interior pages. Predict first: fanout ratio ⇒ height change at 1M keys?

8. Capstone milestone M3 (in ../../capstone/)

• Disk-backed B+tree behind the M1 storage trait: properties + range indexes.
• Design the page format FIRST (no peeking), then compare with the reference cow_btree (in-memory Arc-page COW) — write up: what changes when pages live on disk vs in Arc? (Free-space mgmt, splits, checksums vs refcounts.)
• Range-index smoke bench wired into the workload generator.

Done when

Your B+tree passes the crash-free tests, the redb comparison + prefix-truncation numbers are in notes.md with predictions, and you can draw the slotted page + LMDB commit diagrams from memory.

Modern B-tree techniques: height is the metric, fanout is the lever

Every “B-trees are simple” take dies in Graefe’s ~200-page survey of what production B-trees actually do — compression, latching, logging interactions, bulk loads. Do not read it all. This chapter picks the ~50 pages that matter for this topic and the capstone (budget: 3 h); you’ll come back for more in topics 5 (logging), 8/9 (latching), and 12 (columnar).

Read now (this topic)

Defer (note where, come back later)

• §6 latching & B-link trees → topic 9 (concurrency)
• §7 logging & recovery interplay (fence keys, ghost records) → topic 5
• §8 bulk load / index creation → topic 12/22

The three ideas to extract

1. suffix truncation:  separator between "smith,bob" and "smyth,al"
                       needs only "smy" — interior keys shrink ⇒ fanout grows
                       ⇒ height shrinks ⇒ every lookup saves a page

2. prefix truncation:  page stores common prefix once
   page ["foo/aaa".."foo/zzz"]: header prefix="foo/", cells store "aaa"…

3. normalized keys:    encode (type,collation,composite) into memcmp-able bytes
                       — comparison becomes branch-free byte compare (SIMD-able,
                       topic 17)

Suffix truncation is small enough to write down whole — the point is that a separator is synthetic, so it only has to sort between its neighbors:

#![allow(unused)]
fn main() {
// separator between leaf keys "smith,bob" and "smyth,al" → "smy"
fn shortest_separator(left: &[u8], right: &[u8]) -> Vec<u8> {
    let mut i = 0;
    while i < left.len() && left[i] == right[i] {
        i += 1;                       // skip the shared prefix
    }
    right[..=i].to_vec()              // one byte past divergence: > left, ≤ right
}
// shorter separators ⇒ more fit per interior page ⇒ fanout up ⇒ height down —
// and height is priced in page reads, so EVERY lookup collects the saving
}

Fanout arithmetic to internalize: 4KB page, 16-byte keys + 8-byte child pointers ≈ 170 fanout ⇒ 1B keys in height 4. Truncate separators to 4 bytes ⇒ fanout ~340 ⇒ height 4 still, but at 10B keys. Height is the metric; fanout is the lever; key size is what you control.

Questions to answer in notes.md

1. Why does suffix truncation apply to interior separators but prefix truncation mostly to leaf pages? (Separators are synthetic; leaf keys must be exact.)
2. SQLite/turso do neither. Given SQLite’s design goals (simplicity, robustness, integer rowids as the common key), argue whether that’s the right call.
3. Poor man’s normalized key = SwissTable h2 = skiplist tower = pointer-array-as- filter. Write the general principle in one sentence for the capstone notes.

Done when

You can do the fanout→height arithmetic cold, and you’ve marked which sections you’ll return to in topics 5 and 9.

References

Papers

• Graefe — “Modern B-Tree Techniques” (Foundations and Trends in Databases, 2011) — ~200 pages; do NOT read it all — follow the section table above (§3 truncation + normalized keys and §5 node sizes now; §6–§8 deferred to topics 9, 5, and 12/22)

LMDB: recovery is choosing a root pointer

LMDB is the anti-SQLite: no WAL, no page cache of its own, no free-space- within-page — just copy-on-write pages over one big mmap, with crash recovery reduced to picking the newer of two meta pages. This chapter reads its single 12,846-line file as a design, skimming the code (2 h); it is also the on-disk twin of the capstone reference’s in-memory cow_btree, which is exactly M3’s comparison exercise.

1. The commit protocol — the whole design in one sequence

• Two meta pages at file offsets 0 and 1; txn N writes meta N % 2 (comment mdb.c:1356, MDB_meta struct :1358).
• mdb_txn_commit → mdb_page_flush (write dirty pages) → fsync → mdb_env_write_meta (mdb.c:4847, slot txnid & 1 at :4863) → fsync.

 crash timeline:                              recovery = nothing:
 write pages ─ fsync ─ write meta ─ fsync     open env, read both metas,
      ▲crash: old meta wins   ▲crash: old     pick larger valid txnid
       (new pages unreachable) meta wins      (mdb_env_pick_meta)

No WAL, no redo, no undo. Recovery is choosing a root pointer. The price is paid elsewhere: every commit rewrites the whole root-to-leaf path.

The protocol fits on a napkin:

#![allow(unused)]
fn main() {
fn commit(env: &mut Env, txn: Txn) -> Result<()> {
    write_pages(&txn.dirty)?;               // COW pages at NEW page numbers —
    fsync(env.fd)?;                         //   durable before any root sees them
    let meta = Meta { txnid: txn.id, root: txn.new_root };
    write_meta_slot(env, (txn.id % 2) as usize, &meta)?;  // toggle: never
    fsync(env.fd)                                         //   overwrite live meta
}

fn open(env: &Env) -> Root {
    let (m0, m1) = read_both_metas(env);
    pick_valid_with_larger_txnid(m0, m1).root   // recovery IS this line —
}                                               // a crash anywhere above just
                                                // means the old meta still wins
}

2. COW mechanics

• mdb_page_touch — mdb.c:3015: first write to a clean page in a txn copies it to a fresh page number; the parent’s child pointer is updated (parent was touched first — the descent touches top-down).
• Dirty pages tracked in mt_u.dirty_list (sorted ID list; insert at mdb_page_dirty :2670) — flushed sequentially at commit.
• Compare with the capstone reference’s in-memory cow_btree: same path-copy, but Arc refcounts replace the freelist, and “commit” is an atomic root swap instead of a meta-page write. Write this comparison in notes — it’s M3’s core.

3. Page reuse — GC as a database

• Freed page IDs go into a freelist database (FREE_DBI, mdb.c:1345) keyed by the txn that freed them.
• mdb_page_alloc (mdb.c:2693) reuses freed pages only if freed by a txn older than the oldest active reader (mdb_find_oldest :2640 scans the reader table mti_readers, MDB_reader struct :869 — one slot per reader in a shared lock file, holding a frozen mr_txnid).
• Consequence: a stalled reader pins EVERY page version since its snapshot — the file grows without bound. (The infamous LMDB “long-lived reader” footgun; the reference cow_btree has the same issue as Arc-pinned snapshots.)

4. Readers never block writers

• Read txn: mdb_txn_renew0 (mdb.c:3285) picks the newest meta (mdb_env_pick_meta :3296), records its txnid in a reader slot — that’s the entire read-txn setup. No locks on the data pages, ever.
• Single writer at a time (writer mutex) — LMDB doesn’t pretend otherwise.

5. The mmap

• mdb_env_map — mdb.c:5040: one big PROT_READ mmap; writes go through pwrite (default) or a writable map with MDB_WRITEMAP (:5097).
• Reads = pointer dereference into the map — zero-copy, no buffer pool, the OS page cache IS the cache. Topic 6’s mmap-considered-harmful paper will argue why this is dangerous for writes (no control over write-back order) — note that LMDB’s default mode avoids exactly that by using pwrite + the meta protocol, not the writable map.

6. Search/split (skim)

• mdb_page_search :7535 → mdb_node_search :6689 (binary search per page).
• mdb_page_split :10662 — median promotion, cascading up. Simpler than SQLite’s 3-sibling balance: COW means the path is being rewritten anyway, so there’s no “redistribute in place to avoid dirtying neighbors” incentive.

Questions to answer in notes.md

1. Why does LMDB’s split not bother with SQLite-style sibling redistribution? (COW already dirties the path; also no freeblocks — append-style page builds.)
2. Double meta + fsync ordering: which of the two fsyncs could you drop, under what hardware assumption, and what breaks on consumer SSDs?
3. Price a 1-key commit at tree height 4, 4KB pages: bytes written for LMDB vs a WAL engine (≈ record + fsync). When does LMDB’s model win anyway? (Read-heavy, batch-committed writes.)

Done when

You can narrate a crash at any point in the commit sequence and say which root survives, and you can state the reader-pins-pages problem and its capstone twin.

References

Code

• LMDB libraries/liblmdb/mdb.c (12,846 lines, one file; local clone at ~/repos/lmdb) — read it as a design, skim the code; the MDB_meta comment (:1356) and the reader table (MDB_reader :869) carry the whole model

btree.c: twenty years of production scars

You already know the format from turso — this guided skim (2 h) reads the original for the parts turso simplified and for comments that carry two decades of production experience: the balance_quick fast path, the “25% faster” right-bias tweak, pointer maps, predecessor-swap deletes. Don’t read its 11,633 lines linearly; follow the route below.

1. Start with btreeInt.h:1–215

The file-format spec as a comment: page layout diagram, cell formats, freeblock list, overflow, freelist. This is the best on-disk-format documentation in open source. Read it entire.

Key structs:

• MemPage — btreeInt.h:273–303. Note xCellSize / xParseCell function pointers picked once per page type at init — devirtualized dispatch, 1994 style. And nFree is computed lazily (−1 until needed).
• CellInfo — btreeInt.h:480–486: nKey, pPayload, nLocal, nSize.

2. The search path

• sqlite3BtreeTableMoveto — btree.c:5837–5978. Binary search :5917–5954; child descent :5965–5971 (lwr >= nCell ⇒ rightmost pointer). Note the bias hint parameter — appenders skip the binary search.
• sqlite3BtreeIndexMoveto — btree.c:6068–6295. Uses an xRecordCompare callback specialized per key shape — same devirtualization move.

3. Balance — read for the engineering, not the algorithm

• balance() dispatcher — btree.c:9162–9225.
• balance_quick — btree.c:8039–8150: rightmost-leaf append gets its own path (sequential inserts are THE common case — fillseq from topic 1).
• balance_nonroot — btree.c:8277–8826. NB = 3 at :7552. Find the comment near :8738: the right-bias optimization “makes the database about 25% faster” — a one-line distribution tweak, measured. Topic-0 lesson in the wild.
• balance_deeper — btree.c:9081: root split = tree grows up.

4. Free space within a page

• allocateSpace — btree.c:1846–1944; freeSpace — :1945–2050 (merges adjacent freeblocks!); defragmentPage — :1640–1837.
• Overflow-cell trick: an overfull page keeps up to apOvfl[] cells beside the page (insertCell :7363–7450) rather than reallocating — balance consumes them immediately. The page is never physically overfull on disk.

#![allow(unused)]
fn main() {
// insertCell's trick: a page is never physically overfull
fn insert_cell(page: &mut MemPage, i: usize, cell: Cell) {
    match page.allocate_space(cell.len()) {     // freeblocks → gap → defrag
        Some(off) => page.write_cell(off, i, &cell),
        None => {
            page.ap_ovfl.push((i, cell));       // parked IN MEMORY, beside the page
            // caller must run balance() before the page is released: the
            // balance pool drains ap_ovfl while redistributing ≤3 siblings,
            // so the on-disk format never needs an "overfull" representation
        }
    }
}
}

5. Two things turso doesn’t have (yet)

• Pointer maps (auto-vacuum) — btreeInt.h:653–668, btree.c:1098–1170: a reverse index (page → parent) so vacuum can relocate pages. Costs a ptrmap page every ~⌊usable/5⌋ pages.
• Interior-delete via predecessor swap — btree.c:9873–10050 (:9954 leaf check, :9956 predecessor fetch): interior deletes become leaf deletes + rebalance.

Questions to answer in notes.md

1. fillInCell (btree.c:7106) builds the overflow chain BEFORE the cell is inserted into the page. What crash-safety property makes that ordering safe? (Pages only become durable at commit via pager/WAL — nothing here is.)
2. Why does balance_quick exist when balance_nonroot handles the same case? Estimate the work saved for a fillseq insert (pages touched, cells copied).
3. SQLite computes nFree lazily and validates cells only under SQLITE_DEBUG. What does that say about where btree.c sits on the trust-the-page-vs-verify spectrum, and what’s the corruption story? (PRAGMA integrity_check exists for a reason.)

Done when

You can explain why NB=3 (bounded work per split, adjacent redistribution beats cascading splits) and name the two fast paths (bias hint, balance_quick) that serve sequential inserts.

References

Code

• sqlite — src/btree.c (11,633 lines; don’t read linearly) and src/btreeInt.h (746 lines) — btreeInt.h:1–215 is the best on-disk-format documentation in open source; read that comment entire before any function

The SQLite file format: decode a row by hand

The normative spec for what btree.c writes — and the one document in this topic you read with a hex dump open beside it. After two codebases’ worth of slotted pages, this chapter verifies your mental model against the official text and ends with the exercise that makes the format yours: labeling every byte of one cell in a real database file. ~1.5 h.

Read in this order

1. §1 The database file — 100-byte file header: page size (offset 16), file change counter, freelist head + count (offsets 32–39), schema cookie.
2. §1.6 B-tree pages — the slotted-page spec you now know from two codebases; verify your mental model against the normative text (esp. freeblock rules: min 4 bytes, fragment cap 60).
3. §2 Record format — serial types table. Note types 8/9 (literal 0 and 1 — zero bytes of payload!) and the odd/even text/blob length encoding (n−13)/2 / (n−12)/2.
4. §1.5 Pointer maps, §4.1 WAL vs rollback journal — skim; WAL is topic 5.

Every cell starts with varints, so carry the decoder in your head into the exercise:

#![allow(unused)]
fn main() {
// SQLite varint: 7 bits/byte, BIG-endian (unlike protobuf), max 9 bytes
fn read_varint(buf: &[u8]) -> (u64, usize) {
    let mut v = 0u64;
    for i in 0..8 {
        v = (v << 7) | (buf[i] & 0x7f) as u64;
        if buf[i] < 0x80 {
            return (v, i + 1);        // high bit clear = last byte
        }
    }
    ((v << 8) | buf[8] as u64, 9)     // 9th byte contributes all 8 bits
}
// rowid 500 = 0x83 0x74 → (0b0000011 << 7) | 0b1110100 — find it in the dump
}

The exercise (30 min, do it)

sqlite3 /tmp/t.db "create table t(a integer primary key, b text);
                   insert into t values (1,'hello'),(500,'world');"
xxd /tmp/t.db | head -80

Find by hand, writing offsets in notes.md:

• page size at offset 16 (big-endian u16);
• page 2’s header byte 0x0D (table leaf), cell count, content-area start;
• the two cell pointers, then decode cell 1: payload-size varint, rowid varint (rowid 500 needs 2 bytes — check the 7-bit encoding), record header, serial type for ‘hello’ (text len 5 ⇒ type 2·5+13 = 23).

If you can decode a row from a hex dump, the format is yours.

Questions to answer in notes.md

1. Why does the format store the cell CONTENT area offset in the header instead of deriving it from the cell pointers? (Cheap free-space check: content_start − ptr_array_end without scanning.)
2. INTEGER PRIMARY KEY tables store the key only as the rowid varint — the column itself is NULL in the record. What does this alias buy in bytes/row and what does it forbid? (WITHOUT ROWID tables exist for the other case.)
3. The change counter (offset 24) and version-valid-for (92) — how do they let a reader detect a stale in-memory schema without locks?

Done when

Your notes contain the annotated hex dump with every byte of one cell labeled.

References

Papers

• SQLite team — “The SQLite Database File Format” (official documentation) — https://www.sqlite.org/fileformat2.html — the normative spec for what btree.c writes; read side-by-side with a real database file and a hex dump

Inside the slotted page: freeblocks, overflow, balance

Topic 1’s turso chapter traced the cursor/seek/insert surface; this one descends into the page mechanics that surface glossed over — the freeblock chain, the exact overflow-spill formulas, the resumable balance state machines, varints, and the whole-page freelist. Budget: 2–3 h across core/storage/btree.rs, sqlite3_ondisk.rs, and pager.rs.

1. Slotted page operations

• Header parsing: btree.rs:76–124 (offsets in README §1).
• Free-slot search: btree.rs:7592–7680 — find_free_slot() walks the freeblock chain (each freeblock: 2B next-ptr + 2B size, threaded through the content area). Minimum slot 4 bytes; smaller leftovers become the header’s fragmented_bytes counter.
• Defragment: btree.rs:8273–8444 — fast path when ≤2 freeblocks, slow path compacts everything. Question while reading: what triggers defrag, and why is it correct to move cells but never the pointer array?

The freeblock walk, distilled — first-fit through a linked list threaded through the dead space itself:

#![allow(unused)]
fn main() {
fn find_free_slot(page: &mut Page, need: usize) -> Option<u16> {
    let mut prev = FREEBLOCK_HEAD;           // header bytes 1–2
    let mut off = page.first_freeblock();
    while off != 0 {
        let (next, size) = page.freeblock_at(off);   // 2B next-ptr + 2B size
        if size as usize >= need {
            let rest = size as usize - need;
            if rest < 4 {                    // leftover can't hold a freeblock:
                page.unlink(prev, next);     //   take it all, book the scraps
                page.add_fragmented(rest as u8);      //   (header cap: 60)
                return Some(off);
            }
            page.set_size(off, rest as u16); // carve the tail, keep the block
            return Some(off + rest as u16);
        }
        prev = off; off = next;
    }
    None    // nothing fits: allocate from the middle gap, or defragment
}
}

2. Overflow — the exact SQLite formulas

• Thresholds: btree.rs:9019–9042 — max_local(index) = (usable−12)·64/255 − 23, max_local(table) = usable − 35, min_local = (usable−12)·32/255 − 23.
• Spill rule: sqlite3_ondisk.rs:2130–2148 — keep min_local + (payload − min_local) % (usable − 4) bytes local.
• Chain format: sqlite3_ondisk.rs:951–961 — last 4 local bytes = next overflow page number (0 terminates).
• Why 64/255 and 32/255? Work it out: they bound local payload so a page always fits ≥4 cells — fanout survives fat keys.

3. Balance — the state machines

Turso’s twist on SQLite: balancing is a resumable state machine (IOResult) instead of synchronous recursion, because every page touch may yield for async IO.

• balance_root() — btree.rs:4774–4852: root overflow ⇒ copy root into a new child, root becomes interior pointing at it (tree grows up).
• balance_non_root() — btree.rs:2995–4087: the ≤3-sibling pool-and- redistribute. Sibling pick at :3305–3375 (left preferred, dividers pulled from parent into the pool); redistribution + new-sibling creation at :3430–3680.
• Trigger: insert overflows the page (btree.rs:2903 — split path after split_cell() can’t fit).

 balance_non_root, 2 siblings + overfull page:

 parent:      [ ... D1 ... D2 ... ]         D = divider cells
                 │      │      │
        [sib L]   [OVERFULL]   [sib R]
        └──────── pool: L + D1 + full + D2 + R ────────┘
                     redistribute evenly ⇒ 2–4 pages, new dividers up

4. Varints and the record format

• read_varint / write_varint — sqlite3_ondisk.rs:1304–1336 / 1379–1421: 7 bits/byte big-endian, 9th byte carries a full 8 bits (max 9 bytes for u64).
• Record: header-size varint, then per-column serial type varints, then the values (sqlite3_ondisk.rs:1101–1237). Serial types encode type AND length in one number — schema-less pages.

5. Freelist (whole-page recycling)

• Trunk page: sqlite3_ondisk.rs:89–93 — next-trunk u32, leaf-count u32, then leaf page numbers. Leaves are just free page IDs.
• allocate_page() — pager.rs:5250–5448: pop a leaf; if trunk empty, the trunk page ITSELF becomes the allocated page. add_page_to_freelist() — pager.rs:5101–5145.

6. Cell formats

Table interior child u32 ∥ rowid varint; table leaf size ∥ rowid ∥ payload; index interior child ∥ size ∥ payload; index leaf size ∥ payload (structs sqlite3_ondisk.rs:775–812, parsing :826–930). Note: no prefix/suffix truncation anywhere — turso (like SQLite) stores full keys. That’s your experiment’s opening.

Questions to answer in notes.md

1. Why do table-btree interior cells store only rowids (no payload) while index-btree interior cells carry the full key? What does that do to fanout?
2. The freeblock minimum is 4 bytes and fragmented_bytes caps at 60 in SQLite — what goes wrong without defragmentation? When must allocateSpace defrag even though total free space suffices?
3. Turso’s balance yields mid-operation for IO. What invariant must hold at every yield point so a concurrent reader (or a crash) never sees a broken tree? (Hint: WAL — pages aren’t durable until commit; in-memory the cursor holds refs.)

Done when

You can write the byte layout of a table-leaf page containing two cells and one freeblock, from memory, and explain what balance_non_root pools and why ≤3.

References

Code

• turso — core/storage/btree.rs (slotted-page ops, balance state machines), core/storage/sqlite3_ondisk.rs (overflow, varints, cell formats), core/storage/pager.rs (freelist) — local clone at ~/repos/turso; line numbers drift, navigate by symbol name. Extends topic 1’s reading-turso-btree.md

Topic 3 — notes

Predictions (fill BEFORE running)

Fanout arithmetic check (before measuring): 4KB page, 8B key + 2B ptr + 4B lens ⇒ leaf holds ~N cells; interior fanout ~M ⇒ predicted height at 1e6 keys = ___.

Reading answers

turso deep (reading-turso-btree-deep.md)

1. Table vs index interior cell contents / fanout:
2. Why defrag is needed despite freeblocks:
3. Yield-point invariant in async balance:

SQLite btree.c (reading-sqlite-btree.md)

1. fillInCell overflow-first ordering safety:
2. balance_quick savings for fillseq:
3. Trust-vs-verify position:

LMDB (reading-lmdb.md)

1. Why no sibling redistribution on split:
2. Which fsync could go, on what hardware:
3. 1-key commit cost LMDB vs WAL engine; where LMDB still wins:

Graefe survey

1. Suffix (interior) vs prefix (leaf) truncation asymmetry:
2. Is SQLite right to skip both?
3. The one-sentence dense-filter principle:

File format doc

• Annotated hex dump (paste here):

Experiment findings

• Warm-cache caveat: at 1M keys everything fits in the OS page cache — this benches CPU + page format, not IO. (Buffer pool + cold runs = topic 6.)
• redb comparison, explained in fanout/height terms:
• Truncation result — fanout before/after, height change, lookup delta:

M3 log

• Page format designed before peeking; diffs vs reference cow_btree noted:
• Disk vs Arc-COW writeup (free-space mgmt, splits, checksums vs refcounts):
• Range-index smoke bench in workload generator:

Topic 4 — LSM-Tree Deep Dive

You know the memtable (topic 2) and the B-tree alternative (topic 3). This topic is the rest of the LSM machine: SST anatomy, bloom filters, and compaction — a scheduling problem wearing a storage-engine costume.

Outcomes

By the end you can:

1. Draw an SST from blocks to trailer and explain restart points + prefix truncation.
2. Derive write amplification for leveled vs tiered compaction and check it by measurement on your own mini-LSM.
3. Explain Monkey’s bloom-bit allocation argument in one paragraph.
4. Say when a write stalls in RocksDB and why stalls are load-shedding, not bugs.

1. The lifecycle (the map for everything below)

flowchart LR
    W["write"] --> MT["memtable<br/>(topic 2 skiplist)"]
    MT -- full --> SEAL["sealed memtable"]
    SEAL -- flush --> L0["L0: overlapping runs"]
    L0 -- "compact (pick by score)" --> L1["L1: disjoint, 10x"]
    L1 --> L2["L2: 100x"] --> LN["…Lmax: ~90% of data,<br/>tombstones die here"]

Reads run the same path in reverse: memtable → sealed → L0 (every run!) → one segment per deeper level (disjoint ⇒ binary search by key range). Every skipped disk probe is a bloom filter earning its bits.

2. Inside an SST

 ┌─────────────┬─────────────┬──────┬─────────────┬────────┬─────────┐
 │ data block  │ data block  │  …   │ filter block│ index  │ trailer │
 │ (~4KB, LZ4) │             │      │ (bloom)     │ block  │ /meta   │
 └─────────────┴─────────────┴──────┴─────────────┴────────┴─────────┘
   inside a data block (restart interval 16):
   [FULL key ∥ v][shared=5,rest ∥ v][shared=7,rest ∥ v]…[FULL key]…[restart offsets]
    ▲ binary search over restart points, linear decode between them

Prefix truncation inside blocks (vs topic 3’s B-tree pages which stored full keys) works because blocks are immutable — write once, no in-place updates to break the delta chain. Immutability is the LSM superpower: checksums per block, whole-file bloom filters, compression — all trivial when nothing mutates.

3. Compaction — the actual design space

RocksDB’s leveled score: L0: files/trigger; L1+: level_bytes/target_bytes (compaction_picker_level.cc:229–233) — highest score compacts first. Write stalls are the back-pressure valve: L0 ≥ 20 files ⇒ slowdown, ≥ 36 ⇒ stop (column_family.cc:1019–1043). No stall mechanism ⇒ unbounded compaction debt ⇒ reads degrade forever. Stalls are the honest choice.

 write amp intuition, leveled, ratio T=10:
 a key is rewritten ~once per level it descends through, but each merge into
 level i drags ~T bytes of level-i data per byte of level-(i-1) data:
 WA ≈ T/2 · levels ≈ 5 · log_T(n/memtable)      ← measure this in your mini-LSM

4. Filters: paying DRAM to skip IO

• Classic rule of thumb: 10 bits/key ⇒ ~1% false positives, k≈7 hashes.
• Monkey’s insight: uniform bits/key is suboptimal — a false positive at a big bottom level costs the same IO as one at a tiny upper level, but the bottom level has ~T× more keys per filter bit. Optimal: more bits/key at smaller levels, exponentially decreasing down the tree; same total DRAM, ~2× fewer false positives.
• RocksDB ships bloom (cache-local, FastLocalBloom) and ribbon filters (~30% smaller, slower to build — CPU-for-DRAM trade; filter_policy.cc:658).

5. Code reading (5–7 h)

• lsm-tree crate (the engine under fjall — read it all, it’s small). → reading-lsm-tree.md — An LSM you can read whole: the lsm-tree crate
• RocksDB db/compaction/ + table/block_based/ — the industrial version. → reading-rocksdb-compaction.md — RocksDB compaction: scores, stalls, and the manifest

6. Papers (4–6 h)

• “Monkey: Optimal Navigable Key-Value Store” (SIGMOD ’17). → reading-monkey.md — Monkey: bloom bits where they pay
• “Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging” (SIGMOD ’18). → reading-dostoevsky.md — Dostoevsky: merge lazily, except at the last level
• “RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications” (TODS ’21). → reading-rocksdb-tods.md — RocksDB’s decade: write amp → space amp → CPU
• “Constructing and Analyzing the LSM Compaction Design Space” (VLDB ’21). → reading-compaction-design-space.md — Compaction is four axes, not two strategies

7. Experiments (in experiments/)

Build a mini-LSM (scaffold compiles; todo!() where the learning is). Optionally follow skyzh/mini-lsm alongside — but the point here is the measurement:

1. src/memtable.rs — wrap your topic-2 skiplist (or BTreeMap to start).
2. src/sst.rs — block-based SST writer/reader: 4KB blocks, restart points every 16, per-block xxhash, whole-file bloom (10 bits/key).
3. src/lsm.rs — put/get/scan, flush at 1MB, pluggable compaction: Leveled (ratio 10) and Tiered (K=4).
4. The experiment (src/bin/write_amp.rs): load 10M keys (uniform random overwrite, 3 passes), count bytes written to disk / bytes of user data — write amp per strategy. Also record: read amp (segments probed per get, bloom hits/misses), space amp (dir size / live data). Fill the RUM table with MEASURED numbers.

8. Capstone milestone M4 (in ../../capstone/)

• LSM-backed persistence alternative: graph snapshots/deltas as SSTs behind the M1 storage trait.
• Bench B+tree (M3) vs LSM backends: graph-mutation stream (edge inserts, property updates) and bulk-load; report write amp + p99 with tail-latency discipline (topic 0 rules).
• Note where FalkorDB’s actual persistence (redis RDB/AOF fork-snapshot) sits relative to both — neither B-tree nor LSM; that comparison seeds topic 5.

Done when

Mini-LSM passes tests; the write-amp table (leveled vs tiered, measured vs predicted) is in notes.md; you can explain Monkey’s allocation and the stall triggers without looking.

Compaction is four axes, not two strategies

“Leveled vs tiered” is a false binary: a compaction policy is an independent choice on four design axes — trigger, layout, granularity, movement — and every system you’ve read in this topic sits somewhere in that grid. This is the taxonomy chapter; read it LAST of the four papers, because it organizes the other three.

The four axes (§3 — the contribution)

 a compaction policy = choice on each axis:

 1. TRIGGER      when?    level saturation / #runs / staleness / space amp
 2. DATA LAYOUT  what shape?   leveling / tiering / 1-leveling / L-leveling / hybrid
 3. GRANULARITY  how much at once?   whole level / one file (RocksDB) / few files
 4. DATA MOVEMENT who moves?   full merge / trivial move (relink non-overlapping)

Your mini-LSM: trigger = level size, layout = leveled or tiered, granularity = whole level, movement = full merge (+ trivial move if you stole lsm-tree’s Choice::Move). Locate every system you’ve read on these axes — RocksDB leveled is (saturation, leveling, one-file, merge+trivial-move).

Reading order

1. §3 — the taxonomy. Make the table for: your mini-LSM, lsm-tree crate, RocksDB leveled, RocksDB universal, FIFO.
2. §4 — the benchmark methodology: they implement the design space inside one engine to compare fairly. This is the Fair Benchmarking paper’s lesson (topic 0) applied — same engine, one variable.
3. §5 findings — the ones worth keeping:
   • file-granularity compaction (RocksDB style) smooths write stalls vs whole-level (spikes) — granularity is a tail latency knob, not a throughput knob;
   • trigger choice dominates point-lookup latency more than layout at low write rates;
   • no policy wins everywhere (the RUM conjecture, empirically, again).
4. Skim the workload sensitivity plots — note which finding you’ll test.

Questions to answer in notes.md

1. Your write_amp experiment compacts whole levels. Predict, then measure if time allows: what does per-insert p99.9 look like vs a per-file granularity variant? (This is topic 2’s rehash-spike lesson at LSM scale.)
2. Which axis does Dostoevsky’s lazy leveling move on? (Layout only — trigger/ granularity/movement orthogonal.) Which does Monkey move on? (None — it’s a filter-memory axis the taxonomy doesn’t cover; where would you add it?)
3. For M4’s graph-snapshot SSTs: bulk-loading a snapshot is one giant sorted run. Which axis choices make ingest cheap? (Trivial move into the bottom level — no merge at all.)

Done when

Your notes contain the 5-system × 4-axis table and one prediction you could test with the mini-LSM.

References

Papers

• Sarkar, Papon, Staratzis, Athanassoulis — “Constructing and Analyzing the LSM Compaction Design Space” (VLDB 2021) — §3 taxonomy and §5 findings are the keepers; §4’s one-engine methodology is the Fair Benchmarking lesson applied

Dostoevsky: merge lazily, except at the last level

Monkey optimized the filters; Dostoevsky optimizes the merging itself — by noticing that most of leveled compaction’s work is “superfluous” (the paper’s word). Point lookups and space amp are dominated by the largest level, write amp by the upper levels, so the right move is to tier the top and level the bottom. This chapter builds that argument and the Fluid-LSM dial that generalizes it.

The insight in one table

Merging at different levels buys different things:

Leveled compaction merges eagerly everywhere — most of that work is “superfluous” (the paper’s word). Tiered merges lazily everywhere — cheap writes but the largest level fragments into K runs, wrecking space amp and zero-result lookups.

Lazy Leveling (the contribution)

 tiered:            leveled:            lazy leveled (Dostoevsky):

 L1: ▧▧▧▧ K runs    L1: ▧ 1 run        L1: ▧▧▧▧ K runs   ← tiered on top
 L2: ▧▧▧▧           L2: ▧              L2: ▧▧▧▧             (writes cheap)
 L3: ▧▧▧▧           L3: ▧              L3: ▧ 1 run       ← leveled at bottom
                                                            (space + reads OK)
 WA: O(L)           WA: O(T·L)         WA: O(L + T)  ← T paid once, at bottom

Point lookups and space amp are dominated by the largest level; write amp is dominated by the upper levels (data passes through them repeatedly). So: tier the top, level the bottom. Fluid LSM generalizes with two knobs (K = runs allowed at upper levels, Z = runs at the largest) and picks them per workload — leveled (K=Z=1) and tiered (K=Z=T−1) become endpoints of a dial.

The whole family is one compaction chooser with two thresholds:

#![allow(unused)]
fn main() {
// K = max runs at upper levels, Z = max runs at the largest level.
// K=Z=1 ⇒ leveled; K=Z=T−1 ⇒ tiered; K=T−1, Z=1 ⇒ lazy leveling.
fn choose(&self, v: &Version) -> Choice {
    for lvl in 0..v.last_level() {
        if v.runs(lvl) > self.k {                 // upper levels: tolerate K runs
            return Choice::MergeRunsInto(lvl + 1);
        }
    }
    if v.runs(v.last_level()) > self.z {          // largest level: tolerate Z
        return Choice::MergeLastLevel;            // T paid once, here
    }
    Choice::DoNothing
}
}

Reading order

1. §2 — the cost table (Table 1). Reproduce it for yourself for T=10, L=3: write cost, point read (zero/non-zero result), range, space. This table IS the paper.
2. §3 — Lazy Leveling analysis. Check the claim: same point-read + space complexity as leveled, write cost close to tiered.
3. §4 — Fluid LSM + the tuning section (skim the solver, keep the knobs).
4. Evaluation — find the throughput-vs-skew plots.

Questions to answer in notes.md

1. Your mini-LSM implements leveled and tiered. Using its measured write amp and read amp: on YOUR numbers, what would lazy leveling have scored? (Compute — upper levels tiered cost + bottom leveled cost.)
2. Why do range scans not benefit from lazy leveling the way point reads do? (Every run at every level must be merged into the scan regardless.)
3. RocksDB never shipped lazy leveling as such — universal compaction covers part of the space. From reading-rocksdb-compaction.md, which universal knobs approximate K and Z?

Done when

You can reproduce Table 1 from memory for the three strategies (writes, point reads, space) and say in one sentence why “merge lazily except the last level” dominates.

References

Papers

• Dayan & Idreos — “Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging” (SIGMOD 2018) — §2’s cost table (Table 1) IS the paper; §3 for the lazy leveling analysis, §4 for Fluid LSM (skim the solver, keep the knobs)

An LSM you can read whole: the lsm-tree crate

Every LSM concept in this topic — restart-point block encoding, bloom-gated point reads, versioned level metadata, pluggable compaction — exists as a few hundred readable lines in fjall’s lsm-tree crate. Topic 1 read fjall’s keyspace layer; everything LSM-shaped delegates here, and the crate is small enough to read completely. This chapter orders that read so each layer lands before the one that uses it.

1. Block encoding — src/table/block/

• encoder.rs:61–151 — restart intervals + prefix truncation: a FULL item every restart_interval items; between restarts, items store shared_prefix_len + rest (longest_shared_prefix_length, :142). Optional hash index per block (:148–154) — a tiny SwissTable-ish shortcut inside each block; topic 2 pattern at yet another scale.
• header.rs:49–60 — per-block header: type, xxh3 checksum (u128), on-disk
  • uncompressed sizes; the header itself gets a u32 checksum (:109).
• mod.rs:60–70, 111–120 — LZ4 per block.
• Index blocks: index_block/block_handle.rs:20–43 — varint offset+size handles.

2. Segment writer/reader — src/table/

• writer/mod.rs:40–95 — buffer KVs, flush block at size threshold, feed the filter + index writers, trailer/metadata last (single forward pass — an SST is written append-only, like everything else in an LSM).
• Read path with filter: mod.rs:245–290 — filter loaded lazily; if maybe_contains_hash says no, the point read never touches a data block (:281–288).

3. Bloom filter — src/table/filter/standard_bloom/

• builder.rs:55–127 — with_fp_rate computes m,k from −n·ln(fpr)/ln²2 (:58); with_bpk direct (:93).
• Double hashing — builder.rs:10–13 + mod.rs:102–129: k probes from two hashes via h1 += h2; h2 *= i — k memory probes but only ONE real hash computation. Compare with RocksDB’s cache-local bloom (all k bits in one cache line — topic 0 priced why).

4. Version + levels — src/version/

• mod.rs:42–114 + run.rs:51–103 — levels are runs; a run is disjoint by key range, so get_for_key (:99–103) binary-searches segment ranges: one segment probed per run. L0 = many runs (each flush is one); L1+ = one run.
• Persistence: persist.rs:9–45 — new version file written, checksummed, rewrite_atomic on CURRENT_VERSION_FILE + fsync. This is RocksDB’s MANIFEST in miniature: compaction commits by publishing a new version, never by mutating the old one. COW again, at the metadata level.
• Recovery: recovery.rs:34–95.

5. Compaction — src/compaction/

• Trait: mod.rs:87–98 — choose(version, config, state) → Merge | Move | Drop | DoNothing. Note Move: a segment that doesn’t overlap the next level is relinked, zero IO — find where leveled uses it (leveled/mod.rs:19 pick_minimal_compaction).
• Leveled: leveled/mod.rs:113–143 — L0 trigger 4 runs, ratio 10.
• Worker: worker.rs:382–389 — tombstones evicted only when the output is the last level (evict_tombstones(is_last_level)). Dropping one earlier would resurrect older versions below. Same reasoning you’ll need for M4.
• Merge: merge.rs:35–99 — k-way merge on an interval heap (double-ended, so reverse scans work too).

6. Read path end-to-end — src/tree/mod.rs

• get :639–643 → get_internal_entry :696–750: active memtable → sealed (newest first) → levels; seqno filtering at every step (:701/707/730) — MVCC reads pick the newest version ≤ snapshot seqno.
• Hash computed once and shared across all segment filter checks (:721–723) — the SipHash-cost lesson from topic 0 applied.

The whole path, compressed to its shape:

#![allow(unused)]
fn main() {
fn get(&self, key: &[u8], snapshot: SeqNo) -> Option<Value> {
    if let Some(v) = self.active.get(key, snapshot) { return live(v); }
    for mt in self.sealed.iter().rev() {              // newest sealed first
        if let Some(v) = mt.get(key, snapshot) { return live(v); }
    }
    let h = hash(key);                                // hashed ONCE for all filters
    for run in self.version.runs() {                  // L0: run per flush; L1+: one
        let Some(seg) = run.get_for_key(key) else { continue };  // disjoint ⇒ binary search
        if !seg.filter_maybe_contains(h) { continue; }            // bloom: skip the IO
        if let Some(v) = seg.point_read(key, snapshot) { return live(v); }
    }
    None                                              // live(): tombstone ⇒ None
}
}

• Tombstones: value_type.rs:8–27; hidden at read time (tree/mod.rs:67–72), dropped at bottom-level compaction.

Questions to answer in notes.md

1. Why can L0 not be a disjoint run, and what does that cost a point read? (Flushes overlap arbitrarily ⇒ probe every L0 run ⇒ the stall trigger.)
2. Restart interval 16: derive the trade (space saved by truncation vs linear decode cost per lookup). Why don’t B-tree pages (topic 3) do this?
3. The version file is rewritten whole on every compaction. RocksDB instead appends VersionEdits to a MANIFEST log. When does lsm-tree’s simpler choice break down? (Huge segment counts; crash mid-rewrite handled by atomic rename.)

Done when

You can trace one get from tree/mod.rs:639 to a data-block binary search, naming every filter/index consulted, and explain why tombstones die only at the bottom.

References

Code

• fjall-rs/lsm-tree — the engine under fjall; read it all (~3 h): src/table/block/, src/table/, src/table/filter/standard_bloom/, src/version/, src/compaction/, src/tree/mod.rs. Local shallow clone at ~/repos/lsm-tree.

Monkey: bloom bits where they pay

“10 bits/key everywhere” was folklore; Monkey turned bloom-filter sizing into an optimization problem and won ~2× fewer wasted IOs from the same DRAM. This chapter derives the allocation rule — FPR proportional to level size, so bits/key decrease toward the bottom — and sets up the per-level-bits experiment in the mini-LSM.

The setup

An LSM with L levels, size ratio T, total filter memory budget M. Every level gets a bloom filter. Question: how should M be divided among levels?

The one picture

 uniform (state of practice):        Monkey (optimal):

 L1 (small)  10 bits/key             L1   ~14 bits/key  (FPR tiny)
 L2          10 bits/key             L2   ~12 bits/key
 L3 (huge)   10 bits/key             L3    ~8 bits/key  (FPR larger, but
                                            fewer probes land here anyway)
 total FPR cost: sum of per-level    expected wasted IOs: MINIMIZED —
 FPRs, dominated by... all equally   exponentially decreasing FPR up the tree

Key observation: expected wasted IO for a zero-result lookup = sum of the per-level FPRs (each level is one potential false probe). But bits-per-key buys FPR exponentially (fpr ≈ e^(−bits·ln²2)), while level sizes grow by T. Spending a bit at a small level buys the same FPR drop for T× fewer keys — i.e., T× cheaper. Optimum: FPRs proportional to level size ⇒ bits/key decreasing geometrically toward the bottom; the bottom level may get ~0 (its “filter” is the fact that every lookup ends there anyway).

The whole allocation, as the closed form your mini-LSM can call:

#![allow(unused)]
fn main() {
// Pick a total zero-result FPR budget; hand each level a share
// PROPORTIONAL TO ITS SIZE, then convert fpr → bits/key.
fn monkey_alloc(level_keys: &[u64], total_fpr: f64) -> Vec<f64> {
    let n: u64 = level_keys.iter().sum();
    level_keys.iter().map(|&nk| {
        let fpr = total_fpr * nk as f64 / n as f64;   // p_i ∝ level size
        -fpr.ln() / (LN_2 * LN_2)                     // bits/key: fpr ≈ e^(−bits·ln²2)
    }).collect()                                      // small levels get MORE bits/key
}
}

Reading order

1. §1–2 — the LSM cost model (worth it alone: R/W/M costs as formulas in T, L). Map each symbol to your mini-LSM’s knobs.
2. §4 — the allocation argument (above). Follow the Lagrange-multiplier sketch once; then re-derive the “FPR ∝ level size” conclusion informally yourself.
3. §5 — merging co-tuning (T as a continuum from leveled to tiered). Skim — Dostoevsky does this better.
4. §6 evaluation — look for the ~2× lookup improvement at equal memory.

Questions to answer in notes.md

1. In your mini-LSM (3 levels, T=10, 10M keys), compute uniform-vs-Monkey expected false probes per zero-result get at 10 bits/key average. Then measure zero-result gets both ways (the experiment supports per-level bits-per-key for exactly this).
2. Monkey assumes point lookups dominate. What breaks for range scans? (Filters don’t help ranges at all — prefix blooms exist for a subset.)
3. FalkorDB angle: an attribute store doing existence checks before edge insertion is a zero-result-heavy workload — where would Monkey’s argument apply outside an LSM?

Done when

You can state the allocation rule (“equal marginal IO saved per bit ⇒ FPR proportional to level size”) and back it with the measured table from your mini-LSM.

References

Papers

• Dayan, Athanassoulis, Idreos — “Monkey: Optimal Navigable Key-Value Store” (SIGMOD 2017) — §1–2 for the LSM cost model, §4 for the allocation argument; skim §5 (Dostoevsky does the merging co-tuning better) and §6 for the ~2× lookup improvement at equal memory

RocksDB compaction: scores, stalls, and the manifest

The lsm-tree crate gave you the clean shape; RocksDB is what a decade of production adds on top — score-driven compaction picking, write stalls as back-pressure, partitioned indexes, ribbon filters, and a MANIFEST that does MVCC for metadata. This chapter is a guided skim of exactly those additions.

1. Leveled picking — db/compaction/compaction_picker_level.cc

• Score formulas in comments :229–233 — L0: num_files / level0_file_num_compaction_trigger; L1+: level_bytes / MaxBytesForLevel. Highest score wins.
• LevelCompactionBuilder::PickCompaction :531 — setup inputs, expand to clean key boundaries, grab the overlapping next-level files.
• :596 — score recomputed accounting for in-flight compactions — the picker is a scheduler; double-booking a level wastes IO.

The scoring loop, reduced to its logic:

#![allow(unused)]
fn main() {
// Compact the level with the highest score ≥ 1.0; the picker is a
// scheduler, so bytes already being compacted don't count twice.
fn pick_compaction_level(&self, v: &Version) -> Option<usize> {
    (0..v.num_levels())
        .map(|lvl| {
            let score = if lvl == 0 {
                v.num_l0_files() as f64 / self.l0_file_trigger as f64
            } else {
                (v.level_bytes(lvl) - v.bytes_being_compacted(lvl)) as f64
                    / self.max_bytes_for_level(lvl) as f64
            };
            (lvl, score)
        })
        .max_by(|a, b| a.1.total_cmp(&b.1))
        .filter(|&(_, score)| score >= 1.0)   // below 1.0: no debt, do nothing
        .map(|(lvl, _)| lvl)
}
}

2. The merge itself — compaction_job.cc:1904

ProcessKeyValueCompaction: a k-way merge (like lsm-tree’s) plus production concerns — compaction filters (user callbacks), snapshot lists (which old versions must survive), sub-compaction splitting for parallelism. Skim for shape; the interesting part is how much of it is not the merge.

3. Stalls — db/column_family.cc:1019–1043

GetWriteStallConditionAndCause:

• L0 files ≥ level0_stop_writes_trigger → stop
• pending compaction bytes ≥ hard limit → stop
• L0 files ≥ level0_slowdown_writes_trigger → delayed
• pending bytes ≥ soft limit → delayed

Compaction debt is measured in bytes not yet merged; stalls convert an unbounded read-amp problem into a bounded write-latency problem. Compare fjall’s version: a spin-loop delay at 20–30 L0 runs (fjall src/keyspace/write_delay.rs:8–16) — same valve, 100× simpler.

4. SST building — table/block_based/block_based_table_builder.cc

• Restart interval + delta encoding :1096–1097 (default 16 — same constant as lsm-tree; convergent evolution or shared ancestry? LevelDB is the ancestor).
• Block flush policy :1127 (~4KB).
• Index entry = last key of each block, written on flush :1908–1912; SQLite’s interior separators, rediscovered — and RocksDB DOES shorten them (FindShortestSeparator), the truncation topic 3 experimented with.

5. Read path — block_based_table_reader.cc

• Get :3010 — whole-table filter check first :3040, then index iterator

  3044–3053, then data block :3071–3096, block cache probe in GetDataBlockFromCache :2345.

• Partitioned index :1778 + partitioned_index_reader.h:15 — the index itself becomes a 2-level B-tree when tables are huge: top level pinned in cache, partitions loaded on demand. No fractional cascading in practice — plain binary search per level won.

6. Filters — table/block_based/filter_policy.cc

• FastLocalBloomBitsBuilder :365–376 — millibits_per_key; probes stay within one cache line per key (contrast lsm-tree’s double hashing across the whole bit array — k cache lines).
• Ribbon :658–686 — ~30% smaller for the same FPR, costlier to build; falls back to bloom if banding fails after 256 seed attempts. CPU-for-DRAM knob.

7. The manifest — db/version_set.cc

• LogAndApply :6778 — compaction output replaces inputs by appending a VersionEdit (version_edit.h:37–77, 705–744) to the MANIFEST log, then pointing CURRENT at it. Readers keep iterating their old Version (refcounted) — MVCC for metadata. lsm-tree rewrites the whole version file instead; same atomicity, different scale point.

Questions to answer in notes.md

1. Why does leveled compaction pick by score rather than round-robin? Construct a workload where round-robin lets one level grow unboundedly.
2. Partitioned index vs lsm-tree’s per-block hash index — both attack “index too big for cache”. Which helps point reads, which helps scans, why?
3. FastLocalBloom does k probes in one cache line — what does that cost in FPR vs a classic bloom at equal bits/key? (Blocked blooms have slightly worse FPR — the locality is paid for in statistics.)

Done when

You can list the three stall triggers from memory and explain LogAndApply’s refcounted-Version scheme as “MVCC for metadata”.

References

Code

• facebook/rocksdb — db/compaction/compaction_picker_level.cc, db/compaction/compaction_job.cc, db/column_family.cc (stalls), table/block_based/block_based_table_builder.cc, table/block_based/block_based_table_reader.cc, table/block_based/filter_policy.cc, db/version_set.cc (MANIFEST). Local clone at ~/repos/rocksdb.
• fjall-rs/fjall src/keyspace/write_delay.rs — the 100×-simpler stall valve, for contrast.

RocksDB’s decade: write amp → space amp → CPU

Not a data-structures chapter — a 10-years-of-production one. RocksDB’s development priorities shifted three times in a decade, and every shift was driven by hardware economics rather than better algorithms. Read this for what benchmarks don’t show: the failure modes, API regrets, and configuration sprawl that only appear at fleet scale.

The arc (what to extract)

The paper’s history of what RocksDB optimized for, in order:

2012 ───────► 2015 ───────► 2018 ───────► 2021
write amp     space amp     CPU           disaggregated / remote storage
(SSD wear,    (SSDs got     (storage got  (storage moves off-box;
 fillrandom    cheaper —     fast enough   topic 28 territory)
 benchmarks)   $/GB rules)   that CPU is
                             the bottleneck)

Each shift happened because the hardware economics moved, not because the algorithms improved. Leveled compaction’s high write amp was acceptable the moment space amp mattered more — the RUM triangle steered by procurement.

Read in this order

1. §1–2 — background + the resource-priority history (the arc above).
2. §3 — lessons on compaction: why leveled won at Facebook (space), universal kept for ingest-heavy; the tiered-vs-leveled discussion with production numbers instead of asymptotics.
3. §4 — large-scale lessons. The best section:
   • failure handling: silent corruption found by checksums at every layer (block, file, WAL record) — corruption rates at fleet scale make “unlikely” a certainty;
   • the timestamp/seqno API regrets;
   • configuration sprawl (hundreds of knobs) as an acknowledged failure.
4. §5 — future directions (2021 vintage): remote compaction, tiered storage — check which happened (topic 28 will).

Questions to answer in notes.md

1. The paper says CPU became the bottleneck once NVMe arrived. Reconcile with your topic-0 finding (SipHash 21%, memory stalls dominant): which CPU costs does an LSM add on top of a hash table’s? (Comparisons in merges, block decode/decompress, filter hashing per level.)
2. Why does RocksDB checksum at block AND file AND WAL-record level rather than trusting the filesystem? What’s the FalkorDB/redis equivalent story? (RDB has a CRC; AOF… check.)
3. Pick the lesson from §4 most relevant to the capstone and write one paragraph on how it changes your M4 design.

Done when

You can narrate the write-amp → space-amp → CPU priority arc with the hardware reason for each transition.

References

Papers

• Dong, Kryczka, Jin, Stumm — “RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications” (ACM TODS 2021) — §4 (large-scale lessons) is the best section; §5’s 2021-vintage future directions are checkable predictions for topic 28

Topic 4 — notes

Predictions (fill BEFORE running write_amp)

At 10M ops, 3.3M distinct keys, 100B values, 1MB memtable, ratio 10 / K=4: predicted levels = ___, so:

Reading answers

lsm-tree crate

1. Why L0 can’t be disjoint / what it costs:
2. Restart interval 16 trade; why B-tree pages don’t:
3. Whole-version rewrite vs MANIFEST log breakdown point:

RocksDB

1. Score vs round-robin — adversarial workload:
2. Partitioned index vs per-block hash index:
3. Blocked bloom FPR cost:

Monkey

1. Uniform vs Monkey expected false probes (computed, then measured):
2. What breaks for range scans:
3. Zero-result-heavy workloads outside LSMs:

Dostoevsky

1. Lazy-leveling score on MY measured numbers:
2. Why ranges don’t benefit:
3. Universal-compaction knobs ≈ K and Z:

TODS ’21

1. CPU costs an LSM adds over a hash table:
2. Checksums-at-every-layer; FalkorDB/redis story:
3. The §4 lesson I’m applying to M4:

Design space (VLDB ’21)

• 5-system × 4-axis table:
• Prediction to test (granularity vs p99.9):

Experiment findings

• Measured RUM table above; explain each gap between prediction and measurement:
• Bloom saved __% of probes; at what level did misses concentrate:
• Tombstone test: what compaction bug did the tests catch first try (log it):

M4 log

• LSM backend behind the M1 trait; snapshot-as-SST bulk load uses trivial move:
• B+tree (M3) vs LSM bench on mutation stream + bulk load; p99 discipline:
• Where redis RDB/AOF sits vs both (seeds topic 5):

Topic 5 — Durability: WAL, fsync, Crash Recovery

The hardest part to get right, because the failure you’re defending against deletes the evidence. Four systems, four durability designs: postgres (ARIES- style redo), turso/SQLite (WAL with checksum chain), LMDB (topic 3: no log at all), redis (AOF command log + fork snapshots).

Outcomes

By the end you can:

1. State the WAL rule (log reaches disk before the page it protects) and derive why each system’s recovery works from it.
2. Explain torn pages and the two industrial fixes (full-page writes, double-write).
3. Measure the fsync ladder on your own SSD and design group commit from it.
4. Ship a WAL + crash recovery for your topic-3 B+tree that survives kill -9.

1. The problem and the rule

A 4KB page write is not atomic (power loss mid-sector-train = torn page), and the kernel lies about write completion until fsync. The fix is one invariant:

 WAL rule: log record describing a change is durable BEFORE the changed page.
 commit rule: commit record durable before acknowledging the client.

 write path:            crash recovery:
 1. append log record   1. find last valid log record (checksums!)
 2. fsync log           2. redo forward from last checkpoint
 3. ack client          3. undo losers (if update-in-place; ARIES)
 4. page write LATER       — or skip undo entirely (COW/append-only designs)

2. Four designs on one axis

 no log ◄──────────────────────────────────────────────► log is the database
 LMDB              turso WAL              postgres            redis AOF
 COW + meta flip   frames = page imgs     ARIES redo +        the command
 (topic 3)         appended, checkpoint   FPI after ckpt,     stream itself,
                   moves them home        undo via MVCC       replayed on boot

• turso/SQLite WAL: every commit appends whole page images as frames; a frame with db_size != 0 marks a commit. Reads check the WAL first (page→frame map), the DB file second. Checkpoint = copy frames back. Recovery = scan frames, verify the checksum chain, stop at the last valid commit. No undo, ever — uncommitted frames are simply ignored.
• postgres: logical/physiological records, not page images — except the first touch of each page after a checkpoint writes a full-page image (torn-page defense). Recovery = redo from checkpoint; undo is MVCC’s job (dead tuples), not the log’s.
• redis AOF: log = the commands. appendfsync everysec trades ≤1s of acknowledged writes for throughput — a policy choice the others don’t offer. RDB = fork + COW snapshot (durability by checkpoint only).
• RocksDB WAL: LevelDB record format — 32KB blocks, records fragmented as FULL/FIRST/MIDDLE/LAST with per-fragment CRC (db/log_format.h:22,54, log_writer.cc:87 AddRecord → EmitPhysicalRecord). Block-aligned so a torn tail never hides an earlier record. Memtable + WAL replaces undo entirely.

3. The fsync ladder (measure it — experiment 1)

Group commit exists because of this ladder: if fsync costs 1ms, one fsync per commit caps you at 1K commits/s — but one fsync can cover N commits.

flowchart LR
    A["N threads commit<br/>concurrently"] --> B["leader takes lock,<br/>writes ALL queued<br/>records, fsyncs once"]
    B --> C["followers recheck<br/>flushed-LSN: already<br/>covered? return"]

Postgres does exactly this: XLogFlush rechecks LogwrtResult.Flush after acquiring the lock (xlog.c:2885) — most backends find their work already done.

4. Code reading (5–7 h)

• postgres xlog.c (10K lines — guided skim). → reading-postgres-xlog.md — postgres xlog: reserve-then-copy and the flush recheck
• turso WAL — frame format, checksum chain, checkpoint, recovery. → reading-turso-wal.md — Turso’s WAL: recovery is finding where the log ends
• redis aof.c vs rdb.c — command log vs fork snapshot (the FalkorDB reality today). → reading-redis-aof-rdb.md — Redis AOF & RDB: the command stream is the log

5. Papers (3–5 h)

• Mohan et al., “ARIES” (TODS ’92) — summary first, then selected sections. → reading-aries.md — ARIES: recovery when you escape nothing
• “Aether: A Scalable Approach to Logging” (VLDB ’10) — group commit + log-contention analysis on multicore. → reading-aether.md — Aether: one log, no bottleneck

6. Experiments (in experiments/)

1. src/bin/fsync_ladder.rs (provided, runs now) — measure write/ fdatasync/fsync/F_FULLFSYNC latency on YOUR disk. HdrHistogram; the numbers feed every design decision below.
2. src/wal.rs — WAL for the topic-3 B+tree: record format (LSN, page_no, before/after or page image — your call, justify in notes), CRC per record, group-commit API (commit_many).
3. src/bin/crash_test.rs — crash injection: child process inserts keys, parent kill -9s it at a random moment, reopens, verifies: every acknowledged key present, no torn state, unacked keys either fully in or out. Run 100 rounds.
4. benches/commit_throughput.rs — fsync-per-commit vs group commit (batch 8/64/512) vs appendfsync everysec-style. Plot commits/s vs durability window.

7. Capstone milestone M5 (in ../../capstone/)

• WAL + crash recovery for graph mutations (node/edge/property ops as logical records) behind the storage trait.
• Crash-injection suite (the crash_test harness, pointed at the graph).
• Contrast written up: your WAL vs FalkorDB-on-redis (RDB fork snapshot + AOF command replay) — what’s the durability window of each config?

Done when

100/100 crash rounds pass; the fsync ladder table and commit-throughput plot are in notes.md; you can explain why turso needs no undo and postgres needs no logged undo either (MVCC), while ARIES needs both.

Aether: one log, no bottleneck

On a multicore, the log is ONE shared object every transaction must append to and flush — so how does it not become the bottleneck? Aether’s answer is four independent fixes that compose, and one of them (consolidation arrays) is the ancestor of how postgres inserts WAL today. This chapter maps each fix to its modern descendant.

The four bottlenecks (§1–2)

 txn commits ──► [A] contend on log-buffer insert   (one mutex around append)
             ──► [B] wait for fsync                  (I/O latency per commit)
             ──► [C] hold locks WHILE waiting for [B] (lock contention amplified)
             ──► [D] context switches around the wait

Aether attacks each separately — read the paper as four independent fixes that compose:

1. Early Lock Release (§3)

Release locks at commit-record creation, not commit-record durability. Dependent transactions may read your uncommitted-but-logged data — safe IF they can’t acknowledge before you do (their commit record follows yours in the log, so log order enforces the dependency). Serial log = free dependency tracking.

2. Flush pipelining (§4)

Worker threads never block on fsync: they enqueue the commit and detach, picking up new work; a daemon acks completed commits after the flush lands. Throughput of async commit, durability of sync commit — the cost is latency and a more complex scheduler, not a loss window.

3. Consolidation arrays (§5) — the part that shipped everywhere

The insight: even with group commit, every append still serializes on the buffer mutex. Fix: threads combine their requests before hitting the lock.

 naive:      T1 ─lock─ memcpy ─unlock─ T2 ─lock─ memcpy ─unlock─ T3 …
 consolidated:
   T1,T2,T3 meet in a slot array, add up sizes (CAS, no lock),
   ONE of them acquires the lock, reserves sum(bytes) once,
   each thread memcpys into its own slice IN PARALLEL.

Decouples sequencing (must be serial, make it tiny) from copying (can be parallel, make it so). Postgres’s ReserveXLogInsertLocation (spinlock for 3 arithmetic ops) + 8 parallel insertion locks is this idea in production — read reading-postgres-xlog.md §2 side by side with §5.

The slot dance, in code:

#![allow(unused)]
fn main() {
// Combine appends BEFORE the lock; only sequencing stays serial.
fn append(&self, rec: &[u8]) -> Lsn {
    let slot = self.slots.join();                    // CAS onto an open slot
    let my_off = slot.size.fetch_add(rec.len());     // add my bytes — no lock
    if my_off == 0 {                                 // first in = group leader
        let total = slot.close();                    // no more joiners
        let base = {
            let _g = self.buffer_lock.lock();        // tiny critical section:
            self.reserve(total)                      // ONE reservation for all
        };
        slot.publish(base);
    }
    let base = slot.wait_for_base();
    self.buf_write(base + my_off, rec);              // everyone copies IN PARALLEL
    Lsn(base + my_off)
}
}

Read in this order

1. §1–2 for the bottleneck taxonomy (the table above).
2. §5 consolidation arrays — the durable contribution.
3. §3 ELR — for the argument about log order as dependency tracking.
4. Skim §4 + evaluation (§6): note which fix buys what at which core count.

Questions to answer in notes.md

1. Why does ELR NOT violate durability for the dependent transaction? (Its commit record is behind yours; a crash that loses yours loses its too.)
2. ELR hazard: what if the dependent txn’s result escapes to the user by a channel other than its own commit ack (e.g. a read-only txn that never logs)? This is why real systems mostly didn’t ship it.
3. Consolidation vs postgres’s 8 insert locks: both parallelize the copy — what’s the difference in HOW threads find a slot? (CAS-combining into a shared slot vs hashing onto a fixed lock array; contrast under 8 vs 80 writers.)
4. Which bottleneck does your M5 group-commit design leave unfixed? (Likely A — a single mutex around the WAL buffer is fine at graph-workload commit rates; say at what commits/s it wouldn’t be.)

Done when

You can name the four bottlenecks from memory, sketch a consolidation array, and point at the postgres code that embodies it.

References

Papers

• Johnson, Pandis, Stoica, Athanassoulis, Ailamaki — “Aether: A Scalable Approach to Logging” (VLDB 2010) — ~12 pages; §1–2 for the bottleneck taxonomy, §5 (consolidation arrays) is the durable contribution, §3 for the ELR argument, skim §4 and the evaluation

ARIES: recovery when you escape nothing

Postgres escapes undo via MVCC, SQLite-WAL escapes redo via page images, LMDB escapes logging via COW — ARIES is the recovery method for engines that escape nothing: update-in-place, steal, no-force. It is the most-cited recovery paper and the vocabulary every other design in this topic is defined against; reading it tells you exactly what each escape hatch is worth.

Why read it when postgres/turso don’t do full ARIES

ARIES is the recovery design for update-in-place + steal/no-force engines (InnoDB, SQL Server, Db2). Postgres escapes undo via MVCC; SQLite-WAL escapes redo via page images; LMDB escapes logging via COW. ARIES is what you need when you escape nothing — reading it tells you exactly what those escapes are worth.

Vocabulary (the paper is unreadable without these)

The three passes

        log: …──[ckpt: DPT+ATT]────────────────────────► crash
 1. ANALYSIS   ────────────────►  rebuild DPT/ATT from ckpt forward
 2. REDO       ────────────────►  repeat HISTORY (even losers!) from
                                  min(recLSN in DPT) — page LSN ≥ record LSN ⇒ skip
 3. UNDO       ◄────────────────  roll back losers, writing CLRs;
                                  CLR.undoNext skips already-undone work

Repeating history is the counterintuitive core: redo replays everything, including doomed transactions, to restore the exact crash-moment state — THEN undo runs as ordinary, loggable transaction rollback. This is what makes recovery-during-recovery safe (crash during undo ⇒ CLRs ensure no double-undo).

The three passes, as one function:

#![allow(unused)]
fn main() {
fn recover(log: &Log, ckpt: &Checkpoint) {
    let (dpt, att) = analysis(log, ckpt);          // 1. who was dirty, who was active
    for rec in log.from(dpt.min_rec_lsn()) {       // 2. REDO: repeat history —
        if page_lsn(rec.page_id) < rec.lsn {       //    even losers' updates.
            apply_redo(rec);                       //    pageLSN ≥ recLSN ⇒ skip:
        }                                          //    idempotence by LSN compare
    }
    for txn in att.losers() {                      // 3. UNDO: ordinary rollback,
        for rec in txn.updates_newest_first() {    //    but each undo is LOGGED
            let clr = log.append_clr(rec);         //    as a redo-only CLR
            clr.undo_next = rec.prev_lsn;          //    crash mid-undo? restart
            apply_undo(rec);                       //    resumes at undo_next —
        }                                          //    no double-undo, ever
    }
}
}

Read in this order

1. A summary (above) until the three passes + CLRs feel obvious.
2. Paper §3 (“the problem”): why the naive undo-then-redo attempts fail — the best catalog of recovery bugs ever assembled.
3. §6: the passes in detail — read for pageLSN ≥ recLSN ⇒ skip redo (idempotence via LSN comparison) and the CLR undoNext chain.
4. Skim §10 (nested top actions — how B-tree splits survive rollback: the split stays even if the insert that caused it aborts).

Map to what you’ve read

• postgres: ANALYSIS+REDO yes (xlogrecovery.c), UNDO replaced by MVCC vacuum; FPIs make redo idempotent even without perfect LSN discipline.
• turso WAL: no passes at all — commit boundary detection only.
• Your M5 WAL: if you chose logical records (reading-turso-wal.md Q3), you owe ARIES-style idempotent redo: stamp pages with LSNs, skip if page is newer.

Questions to answer in notes.md

1. Why must CLRs be redo-only (never undone)? Walk a crash-during-undo.
2. Nested top action for a B-tree split: why is letting the split survive an aborted insert both correct and necessary? (Other txns may already use the new page; physical consistency ≠ logical visibility.)
3. Which of steal/no-force does YOUR topic-3 B+tree + WAL implement? Derive which passes your recovery needs. (Likely no-steal/no-force at first ⇒ redo-only — say so explicitly.)

Done when

You can fill the 2×2 steal/force matrix with (undo?, redo?) from memory and explain repeating history in two sentences.

References

Papers

• Mohan, Haderle, Lindsay, Pirahesh, Schwarz — “ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging” (ACM TODS 1992) — 70 pages; read a summary first (Franklin’s “Crash Recovery” chapter or CMU 15-445 recovery notes), then dip into §3 and §6, skim §10

postgres xlog: reserve-then-copy and the flush recheck

Postgres’s WAL is 10,000+ lines of C, but it earns its keep with five mechanisms: the back-linked record format, the reserve-then-copy insertion trick, group commit via a flush recheck, full-page writes after checkpoints, and fuzzy checkpointing with redo-only recovery. This chapter skims exactly those five — do NOT read the file linearly.

1. The record — xlogrecord.h:41–53

XLogRecord: xl_tot_len, xl_xid, xl_prev (back-pointer — the log is a backward-linked list; recovery validates forward, xl_prev catches missequenced segments), xl_crc. Block references (:103) attach page references; full-page images ride in XLogRecordBlockImageHeader (:141) — note hole_offset: the free space in the middle of a page is elided from the FPI.

2. Insertion — the scalability trick

• ReserveXLogInsertLocation — xlog.c:1149–1193: a spinlock held for ~3 arithmetic ops (:1172–1180) hands out byte ranges in the log. Reservation is serial (tiny), copying is parallel.
• CopyXLogRecordToWAL — xlog.c:1266: copy into the reserved slice of WAL buffers under one of NUM_XLOGINSERT_LOCKS = 8 (xlog.c:157) insertion locks.
• This is Aether’s insight shipped: separate sequencing from copying. Compare topic-2’s incremental rehash — same move, amortize/parallelize the heavy part, keep the critical section O(1).

3. Group commit — XLogFlush, xlog.c:2800–2891

The heart: after acquiring the write lock, recheck LogwrtResult.Flush (:2885) — another backend probably flushed past your LSN while you waited; return without an fsync. commit_delay/commit_siblings (:2901–2906) add an optional pre-flush sleep to grow the batch. Your experiment reimplements this.

Group commit is just that recheck:

#![allow(unused)]
fn main() {
fn xlog_flush(&self, upto: Lsn) {
    if self.flushed_lsn() >= upto { return; }   // cheap check, no lock
    let _g = self.write_lock.lock();            // maybe wait behind a flusher…
    if self.flushed_lsn() >= upto { return; }   // …RECHECK: their fsync already
                                                // covered our LSN — free ride
    self.write_out_buffers_through(upto);
    self.wal_file.fdatasync();                  // ONE fsync for every backend
    self.advance_flushed_lsn();                 // that queued behind the lock
}
}

4. Full-page writes — xloginsert.c:621–700

XLogRecordAssemble: needs_backup = (page_lsn <= RedoRecPtr) (:694) — first modification of a page after a checkpoint logs the whole page (hole elided). Cost: WAL volume spikes after every checkpoint (the famous sawtooth). This is the torn-page defense; InnoDB solves the same problem with a double-write buffer instead; LMDB and SQLite-WAL solve it by never overwriting.

5. Checkpoint + recovery

• CreateCheckPoint — xlog.c:7400–7560: set redo point under the insert lock (:7561), then flush dirty buffers while WAL keeps rolling — fuzzy: the checkpoint is a starting point, not a consistent snapshot.
• PerformWalRecovery — xlogrecovery.c:1612–1806: read from the redo point, ApplyWalRecord (:1782) dispatches to per-resource-manager redo handlers. Per-record CRC in xlogreader.c:1207–1227 — an invalid CRC means “end of log”, not “corruption error”. The log’s tail is expected to be garbage after a crash; checksums are how you find the cliff edge.
• No undo pass: postgres MVCC never overwrites tuples in place, so losers are just dead tuples awaiting vacuum. ARIES’s undo machinery (CLRs, rollback) isn’t needed. Read reading-aries.md for what postgres is not doing.

6. Sync methods — issue_xlog_fsync, xlog.c:9361–9410

wal_sync_method: fsync / fdatasync / open_datasync (O_DSYNC at open — no separate sync call). Your fsync_ladder experiment measures exactly these.

Questions to answer in notes.md

1. Why is xl_prev needed when records are read forward anyway? (Detects a valid-looking record left over from a recycled segment file.)
2. FPI sawtooth: checkpoint_timeout ↑ ⇒ WAL volume ↓ but recovery time ↑. Write the trade as a formula in (dirty rate, checkpoint interval).
3. The 8 insertion locks: what workload would make you raise the number, and what does postgres pay for each extra lock at flush time? (Flush must wait for all in-progress copies below the target LSN — WaitXLogInsertionsToFinish.)

Done when

You can explain reserve-then-copy, the flush recheck, and needs_backup in three sentences total — those three lines are the file.

References

Code

• postgres/postgres — src/backend/access/transam/xlog.c (10,196 lines — do NOT read linearly), src/backend/access/transam/xloginsert.c, src/backend/access/transam/xlogrecovery.c, src/include/access/xlogrecord.h. Local clone at ~/repos/postgres.

Redis AOF & RDB: the command stream is the log

Redis logs the commands themselves (AOF) and checkpoints by forking (RDB) — and since a graph module’s data lives inside redis’s keyspace, this is the durability FalkorDB actually has today. Read it as the incumbent your M5 design competes with: what everysec’s ack-before-durability really promises, and why AOF rewrite is an LSM compaction in disguise.

1. AOF = the command stream is the log

• feedAppendOnlyFile — aof.c:1409–1448: every write command appended (as RESP text!) to server.aof_buf (:1444).
• flushAppendOnlyFile — aof.c:1147–1355: buffer → write() (:1218), then the policy (:1330–1354):
  • AOF_FSYNC_ALWAYS (:1337) — fsync before ack. Durable, slow.
  • AOF_FSYNC_EVERYSEC (:1350) — fsync on the bio background thread (aof_background_fsync, :983): main thread never blocks on the disk; window = up to ~2s of acked writes.
  • AOF_FSYNC_NO — kernel decides. Window = unbounded.
• Group commit comparison: everysec is group commit with a time batch and the ack BEFORE the flush — postgres groups the flush but never acks early. Different contract, not just different tuning.

The three contracts, side by side:

#![allow(unused)]
fn main() {
// flushAppendOnlyFile: the client was ACKED before any of this runs.
fn flush_aof(&mut self, policy: Fsync) {
    self.file.write_all(&self.aof_buf);          // into page cache only
    self.aof_buf.clear();
    match policy {
        Fsync::Always => self.file.fdatasync(),  // durable before next ack: slow
        Fsync::EverySec => {
            if self.last_fsync.elapsed() >= Duration::from_secs(1) {
                self.bio.submit(FsyncJob);       // background thread — main
            }                                    // thread never touches the disk;
        }                                        // window: up to ~2 s of ACKED writes
        Fsync::No => {}                          // kernel decides; window unbounded
    }
}
}

2. AOF rewrite — compacting a command log

A command log grows without bound (1M INCRs = 1M records for one key).

• rewriteAppendOnlyFileBackground — aof.c:2652–2720: fork (:2689); the child serializes current state as a fresh BASE file; the parent keeps serving and accumulates new commands into a new INCR file.
• Multi-part AOF (aof.c:45–71): a manifest lists BASE + INCR files — recovery = load BASE, replay INCRs. This is an LSM in disguise: BASE = the bottom level, INCR = L0, rewrite = full compaction, manifest = MANIFEST. (Topic 4’s vocabulary transfers wholesale.)

3. RDB — checkpoint by fork

• rdbSaveBackground — rdb.c:1859–1892: fork (:1868), child walks the keyspace and writes the snapshot; parent’s writes COW pages away. CRC64 trailer (rdb.c:1702–1706).
• The COW cost is why topic-2’s dict disables rehashing during BGSAVE (dict.c:1655) — a rehash would touch every bucket and copy the whole table. Durability policy reaching down into data-structure design.

4. The FalkorDB angle (write this up in notes)

A graph module’s data lives inside redis’s keyspace, so its durability is this file: RDB serializes matrices via module callbacks; AOF logs the GRAPH.QUERY commands. Questions that matter for M5:

• Replaying GRAPH.QUERY commands re-executes parsing and planning — what’s recovery time for 10M mutations vs replaying logical records?
• An RDB snapshot of a multi-GB graph forks + COWs the whole matrix set under write load — measure-or-estimate the stall.

Questions to answer in notes.md

1. everysec acks before durability. State the exact loss window and why redis considers delaying writes (not acks) when the bio fsync falls behind (:1147 area — the “postpone” logic).
2. AOF-as-LSM: map BASE/INCR/rewrite/manifest onto topic-4 terms. What’s the “write amp” of an AOF rewrite?
3. Command-log vs page-image vs logical-record WAL: rank recovery speed and log volume for a graph-mutation workload; justify your M5 choice.

Done when

You can state each appendfsync policy’s durability window from memory and explain AOF rewrite as compaction.

References

Code

• redis — src/aof.c (feed, flush policies, rewrite, multi-part manifest) and src/rdb.c (fork + COW snapshot, CRC64 trailer). Local clone at ~/repos/redis.

Turso’s WAL: recovery is finding where the log ends

This is SQLite’s WAL mode in Rust: commits append whole page images as frames, a chained checksum makes the log’s valid prefix self-evident, and recovery has no redo or undo at all — it just decides where the log ends. Of the four durability designs in this topic, this is the one your experiment should steal most from.

1. The format — page images, not operations

• WalHeader — sqlite3_ondisk.rs:411–443: magic, version, two salts, header checksum.
• WalFrameHeader — :477–495: 24 bytes — page_no, db_size (non-zero ⇒ this frame commits), the salts (copied from the header), checksum_1/2.
• Frame write — :2058–2090: checksums are cumulative: each frame’s checksum seeds the next (:2080–2088). One flipped bit invalidates the entire suffix — exactly what you want: recovery can trust everything before the break.

 WAL file:  [hdr]  [frame p5][frame p2][frame p9*]  [frame p5][frame p1*] …
                    ── txn 1, *commit (db_size≠0) ──  ── txn 2 ──
 checksum:   c0 ──► c1 ──────► c2 ─────► c3 ─────────► c4 ──────► c5   (chained)
 salts: change on WAL reset — a stale frame from a previous WAL generation
        fails the salt check even if its checksum chain looks plausible

2. Commit + sync

• prepare_wal_finish — wal.rs:4130–4145: fsync after the commit frame; FileSyncType (io/mod.rs:128–134) distinguishes Fsync from FullFsync (macOS F_FULLFSYNC) — on your Mac, plain fsync does NOT flush the drive cache. Your fsync_ladder experiment will show the gap; it’s not subtle.

3. Reads check the WAL first

• find_frame — wal.rs:3335–3404: in-memory page→latest-frame map; hit ⇒ read_frame (:3409) from the WAL file; miss ⇒ read the main DB.
• Consequence: an uncheckpointed WAL makes every read do a map lookup, and a HUGE WAL makes reads slower (more frames to cover) — checkpointing is a read optimization, not just space reclamation.

4. Checkpoint — moving frames home

• CheckpointMode — wal.rs:160–183: Passive / Full / Restart / Truncate.
• checkpoint_inner — wal.rs:4594–4672: backfill loop copies frames [nbackfills+1 … max_frame] into the DB file, sorted by frame for locality.
• Restart/Truncate change the salts — that’s how old frames die without being erased.

5. Recovery — find the cliff edge

• WalScan — sqlite3_ondisk.rs:1426–1932: validate header checksum (:1727), then walk frames verifying salt + chained checksum (:1830–1831); remember the last frame with db_size > 0 (:1844–1855); final state = last valid COMMIT (:1923), not last valid frame — a half-written transaction’s frames are present but unreachable.

No undo, no redo logic — recovery is deciding where the log ends. Compare postgres (redo required: its log holds deltas, not page images) and LMDB (nothing at all: the meta flip made commit atomic).

The whole recovery, in one loop:

#![allow(unused)]
fn main() {
// Walk frames; the answer is the last valid COMMIT, not the last valid frame.
fn recover(frames: &[Frame], hdr: &WalHeader) -> u64 {
    let mut c = hdr.checksum;
    let mut last_commit = 0;
    for (i, f) in frames.iter().enumerate() {
        if f.salts != hdr.salts { break; }     // stale frame from an old WAL generation
        c = chain(c, f);                       // cumulative: one bad bit ends the log
        if c != f.checksum { break; }          // torn frame ⇒ the cliff edge
        if f.db_size != 0 {                    // commit frame
            last_commit = i as u64 + 1;        // a half-written txn's frames stay
        }                                      // present but UNREACHABLE
    }
    last_commit
}
}

Questions to answer in notes.md

1. Why do frames carry whole page images instead of deltas? Name the two things this buys (no redo logic; torn-page immunity — a torn frame fails its checksum and everything after is discarded) and the one it costs (WAL volume ∝ pages touched, not bytes changed).
2. Why salts AND checksums? Construct the failure that checksums alone miss. (WAL reset reuses the file; an old frame at the right offset can have a valid internal checksum — but chains from stale salts.)
3. For your experiment’s WAL: page images or logical records? Decide and justify with the M5 workload (small graph mutations ⇒ logical records win on volume, but then you owe idempotent redo — LSN-stamped pages).

Done when

You can narrate recovery over a WAL containing a torn frame mid-transaction and a complete-but-uncommitted transaction, and say what survives (everything up to the last valid commit frame; both damaged suffixes vanish).

References

Code

• tursodatabase/turso — core/storage/wal.rs, core/storage/sqlite3_ondisk.rs, core/io/mod.rs. Local clone at ~/repos/turso.

Topic 5 notes — durability, WAL, crash recovery

Predictions (fill BEFORE running fsync_ladder)

Predicted max commits/s at 1 fsync/commit: ______ Predicted group-commit speedup at batch 64: ______

fsync_ladder results

(paste table from cargo run --release --bin fsync_ladder)

Surprises vs predictions:

WAL design decisions (src/wal.rs)

• Page images vs logical records — chose: ______ because:
• Group-commit trigger (size / time / both): ______
• Why replay needs no LSN-idempotence check here (and when it would):

crash_test log

• Rounds passed: ___/100
• Failures seen while developing (torn tail? lost ack? partial txn?) and the bug behind each:

commit_throughput results

Reading-guide questions

postgres xlog (reading-postgres-xlog.md)

1. Why xl_prev when reading forward:
2. FPI sawtooth formula in (dirty rate, checkpoint interval):
3. Raising NUM_XLOGINSERT_LOCKS — when, and the flush-time cost:

turso WAL (reading-turso-wal.md)

1. Page images vs deltas — two buys, one cost:
2. The failure salts catch that checksums alone miss:
3. My experiment’s format choice + justification:

redis AOF/RDB (reading-redis-aof-rdb.md)

1. everysec loss window + the write-postpone logic:
2. AOF-as-LSM mapping + rewrite write amp:
3. Command-log vs page-image vs logical-record ranking for graph mutations:

ARIES (reading-aries.md)

1. Why CLRs are redo-only (crash-during-undo walkthrough):
2. Nested top action for a B-tree split — why correct AND necessary:
3. My topic-3 B+tree + WAL: steal? force? ⇒ which passes needed:

Steal/force 2×2 (from memory):

Aether (reading-aether.md)

1. Why ELR preserves durability for dependents:
2. The ELR hazard (non-logging escape channel):
3. Consolidation array vs postgres’s 8 insert locks:
4. Which bottleneck my M5 design leaves unfixed, and at what commits/s it bites:

M5 log (capstone)

• WAL + recovery for graph mutations behind the storage trait
• crash_test harness pointed at the graph — rounds: ___/100
• Contrast vs FalkorDB-on-redis: durability window of RDB-only, RDB+AOF everysec, AOF always:

Topic 6 — Buffer Pool & Memory Management

Who decides which pages live in RAM? Four answers: postgres (CLOCK over a shared array), DuckDB (approximate-LRU queue with lazy purging), LeanStore (pointer swizzling — no mapping table at all on the hot path), and mmap (let the kernel decide — the CIDR ’22 paper on why that’s usually wrong). Plus redis’s answer to a different question: not page caching but allocator accounting (zmalloc + jemalloc + active defrag).

Outcomes

By the end you can:

1. Walk a page request through hash-lookup → pin → CLOCK victim search and say where every atomic and lock is.
2. Explain why mmap loses to a buffer pool (TLB shootdowns, no write ordering, page-fault stalls you can’t schedule around) and when it’s fine.
3. Explain pointer swizzling and the cooling stage — how LeanStore makes an in-memory hit cost ~0 extra instructions.
4. Build a CLOCK buffer pool for your topic-3 B+tree and beat mmap on a larger-than-RAM workload (or measure exactly why you don’t).

1. The translation problem

Every buffer pool is a map page_id → frame, and every design is a stance on who pays for the translation:

                     lookup cost per hot-page hit
 hash table (postgres, DuckDB)     ~1 hash probe + partition lock/atomics
 swizzling  (LeanStore)            0 — the parent's pointer IS the frame ptr
 page table (mmap/OS)              0-ish until a TLB miss / minor fault
                                   ... then the kernel takes over your latency

flowchart LR
    A["request page P"] --> B{"in pool?"}
    B -- "hit" --> C["pin (CAS refcount++)<br/>usage_count↑"]
    B -- "miss" --> D["find victim:<br/>CLOCK sweep"]
    D --> E{"victim dirty?"}
    E -- "yes" --> F["write it out FIRST<br/>(WAL rule: log already flushed)"]
    E -- "no" --> G["evict, relabel frame,<br/>read P from disk"]
    F --> G
    G --> C

2. Eviction: three shapes of approximate-LRU

• postgres CLOCK — one nextVictimBuffer atomic ticks around a fixed array; each buffer has a 4-bit usage_count (max 5). Sweep decrements; a buffer survives up to 5 laps. Pinned buffers are skipped. No linked lists, no per-hit list surgery — a hit is just a saturating increment.
• DuckDB eviction queue — unpinning enqueues (weak_ptr, seq_num) into a concurrent FIFO. Re-pinning doesn’t remove the entry (too expensive); it bumps the handle’s sequence number so the stale entry becomes a dead node, purged in bulk every 4096 insertions. Approximate LRU where the cleanup is amortized, not per-op — same move as topic 2’s incremental rehash.
• LeanStore cooling stage — no global order at all: random buffer frames get unswizzled into a cooling FIFO (~10% of pool). A cool page touched again is re-swizzled cheaply (second chance); reach the FIFO’s end and you’re evicted. Randomness replaces bookkeeping.

3. Pointer swizzling (LeanStore) in one diagram

 swip = one u64 in the PARENT node          (bit 63: evicted, bit 62: cool)
 ┌─────────────────────────────────────────────────────────┐
 │ HOT      00…pointer…   direct BufferFrame* — deref it   │
 │ COOL     01…pointer…   frame in cooling FIFO — CAS back │
 │                        to HOT, done (no I/O, no map)    │
 │ EVICTED  10…page id…   page fault: alloc frame, read,   │
 │                        swizzle pointer                  │
 └─────────────────────────────────────────────────────────┘

Consequence: a page can only be referenced by ONE parent (else two swips to re-swizzle) — fine for B-trees, awkward for arbitrary graphs. Worth pondering for the capstone: matrix blocks form a tree of tiles, so swizzling applies.

4. Why mmap is (usually) wrong — the CIDR ’22 checklist

1. No write ordering — the kernel flushes dirty pages whenever; WAL’s “log before page” needs msync gymnastics or is simply unenforceable.
2. TLB shootdowns — evicting a page means IPIs to every core that might cache the mapping; scales worse with more cores.
3. Page-fault stalls — a fault blocks the thread; no async I/O, no admission control, no prefetch you control.
4. Error handling — I/O errors arrive as SIGBUS mid-instruction.

But: LMDB (topic 3) ships on mmap happily — read-mostly, single-writer, COW keeps ordering trivial. The paper’s “usually” is doing real work. vmcache (SIGMOD ’23) is the synthesis: virtual memory assisted — mmap the address space, but the DB keeps explicit control of residency and eviction.

5. redis: the other memory management

No pages — redis manages allocations. zmalloc wraps jemalloc with per-thread cache-line-aligned used_memory counters (the maxmemory enforcement input), and active defrag literally re-allocates values whose jemalloc bins are underutilized and updates every pointer. FalkorDB’s matrices live inside this world: GraphBLAS blocks are zmalloc’d, counted against maxmemory, and opaque to redis defrag.

6. Code reading (5–7 h)

• postgres bufmgr.c + freelist.c — packed atomic state, CLOCK, buffer rings. → reading-postgres-bufmgr.md — postgres bufmgr: a buffer’s life in one atomic word
• DuckDB buffer manager — eviction queue with dead nodes, memory reservations, spill-to-temp. → reading-duckdb-buffer.md — DuckDB’s buffer pool: eviction by queue of hints
• LeanStore — swips, cooling stage, hybrid latches. → reading-leanstore.md — LeanStore in code: swips, cooling, hybrid latches
• redis zmalloc.c (+ turso’s CLOCK page cache as a bonus). → reading-redis-zmalloc.md — zmalloc: memory management when there are no pages

7. Papers (4–6 h)

• “Are You Sure You Want to Use MMAP in Your DBMS?” (CIDR ’22). → reading-mmap-paper.md — mmap is not a buffer pool
• “LeanStore: In-Memory Data Management Beyond Main Memory” (ICDE ’18) + vmcache (SIGMOD ’23) as the sequel. → reading-leanstore-paper.md — LeanStore & vmcache: pay only on the miss

8. Experiments (in experiments/)

1. src/buffer_pool.rs — CLOCK buffer pool: fixed frame array, page_id → frame map, pin/unpin with usage counts, dirty-page write-back on eviction. Tests fix the contract (pinned pages never evicted, dirty pages written before reuse, capacity respected).
2. src/bin/pool_vs_mmap.rs — same random-read workload over a file 4× larger than the pool/RAM budget: your pool vs mmap. HdrHistogram — compare p50 AND p99.9 (the mmap story is in the tail).
3. benches/eviction.rs — CLOCK vs strict-LRU (linked list) vs FIFO on Zipf-skewed access: hit rate AND ns/lookup. Shows why nobody ships strict LRU (per-hit list surgery costs more than the hit-rate gain).

9. Capstone milestone M6 (in ../../capstone/)

• Buffer pool under the persistent backends — graphs larger than RAM.
• Decide: per-backend pools or one shared pool with MemoryTag-style accounting (DuckDB)? Write the tradeoff down.
• Reproduce mmap write-back unpredictability once, on your Mac, with numbers in notes.

Done when

Your pool beats mmap at p99.9 on the larger-than-RAM benchmark (or you can explain the exact kernel behavior that prevented it); the eviction bench table is in notes.md; you can explain a swip and a dead node from memory.

DuckDB’s buffer pool: eviction by queue of hints

The interesting contrast with postgres: no fixed frame array, no CLOCK — blocks are heap-allocated, tracked by shared_ptr, and eviction is a concurrent FIFO queue of hints that are allowed to go stale. Re-pinning never removes a queue entry; it invalidates one, and dead nodes get swept in bulk. Mark now, collect later — the amortization move again, this time inside the replacement policy itself.

1. BlockHandle — the unit of residency

• block_handle.hpp: BlockState (BLOCK_LOADED/BLOCK_UNLOADED, :62–71), atomic readers pin count (:73–87), CanUnload (:208).
• A BufferHandle (RAII) holds a pin; destruction decrements readers and the block becomes evictable. Rust translation: this is exactly a guard object — your buffer pool’s PageGuard should work the same way.

2. The eviction queue — buffer_pool.cpp

• BufferEvictionNode — :42: a weak_ptr to the block memory + the handle_sequence_number at enqueue time.
• Unpin ⇒ BufferPool::AddToEvictionQueue — :271: bump the handle’s eviction sequence number, enqueue a fresh node; the OLD node for this block (still in the queue!) is now a dead node (:284, IncrementDeadNodes).
• Eviction — EvictBlocks/EvictBlocksInternal (:377+): IterateUnloadableBlocks pops nodes; a node whose seq_num ≠ the handle’s current one is dead — skip; whose weak_ptr won’t lock — dead; else Unload (:38 in that loop) frees the memory.
• Cleanup is amortized: PurgeIteration (:104 hpp) runs every INSERT_INTERVAL = 4096 insertions (:116) and bulk-removes dead nodes.

 re-pin doesn't REMOVE the queue entry (that needs a lock or O(n) search);
 it INVALIDATES it with a seq bump and re-enqueues later.
 → same amortization move as topic 2's incremental rehash and topic 4's
   tombstones: mark now, collect in bulk later.

The eviction loop is mostly corpse-skipping:

#![allow(unused)]
fn main() {
fn evict_until(&self, needed: usize) -> bool {
    let mut freed = 0;
    while freed < needed {
        let Some(node) = self.queue.pop() else { return false };
        let Some(block) = node.block.upgrade() else { continue };  // weak_ptr: block
                                                                   // already gone
        if node.seq != block.eviction_seq.load() { continue; }     // DEAD: re-pinned
                                                                   // since enqueue
        if !block.can_unload() { continue; }                       // pinned right now
        freed += block.unload();          // write to temp file if no disk home
    }
    true
}
}

3. Memory reservations — standard_buffer_manager.cpp

• EvictBlocksOrThrow — :126: every allocation first evicts until the reservation fits, else throws “could not allocate block of size…” (:155). Memory accounting is a gate in front of malloc, not an after-the-fact counter — compare redis, which counts after and evicts keys asynchronously.
• Pin — :333/:337: loaded ⇒ readers++; unloaded ⇒ reserve memory (evicting), reload from disk or temp file.
• Multiple queues by buffer type — buffer_pool.hpp:116–122 (EVICTION_QUEUE_TYPES, priority order): managed buffers vs external files don’t compete in one queue.

4. Spilling — WriteTemporaryBuffer, standard_buffer_manager.cpp:501

Evicted temporary data (hash tables, sorts — no disk home) goes to the temp file manager (:508). This is why DuckDB joins bigger than RAM work: the buffer pool doubles as the spill mechanism. Postgres spills per-operator (work_mem) instead — two philosophies of the same fallback.

Questions to answer in notes.md

1. Why weak_ptr in the queue node? What breaks with shared_ptr? (Queue would keep every block alive — the cache becomes a leak.)
2. Dead-node ratio: worst-case queue length for a workload that re-pins the same block N times between purges. When is CLOCK’s fixed array strictly better?
3. DuckDB throws on memory pressure; postgres errors only when all buffers are pinned. Trace where each behavior comes from and which your capstone pool should adopt (server vs embedded assumptions).

Done when

You can explain a dead node, the 4096-insert purge cadence, and why re-pin never touches the queue — and name the postgres structure each replaces.

References

Code

• duckdb/duckdb — src/storage/buffer/buffer_pool.cpp, src/storage/standard_buffer_manager.cpp, src/include/duckdb/storage/buffer/buffer_pool.hpp, src/include/duckdb/storage/buffer/block_handle.hpp. Local clone at ~/repos/duckdb.

LeanStore & vmcache: pay only on the miss

Two papers, one arc: how to make a buffer-managed system as fast as an in-memory one. LeanStore (ICDE ’18) eliminates the per-access costs with pointer swizzling, a cooling stage, and optimistic latches; vmcache (SIGMOD ’23), from the same group, is “what we’d do differently five years later” — same goal, mechanism moved into the MMU. Read LeanStore first, then vmcache as the retraction-and-fix.

LeanStore: the problem statement (§I–II)

A traditional buffer pool costs a hash lookup + latch + pin per page access even when everything is in RAM. In-memory systems (HyPer) skip all of it. LeanStore’s goal: pay for translation and replacement only on misses — the hot path should look like an in-memory system.

Three ingredients:

 1. pointer swizzling   translation cost → 0    (parent holds raw pointer)
 2. cooling stage       replacement cost → 0    (no per-access bookkeeping;
                        random candidates + second-chance FIFO)
 3. optimistic latches  pinning cost → 0        (readers validate versions,
                        hold nothing)

You’ve read the code (reading-leanstore.md) — in the paper focus on:

• §III.B: the one-swip-per-page ownership rule and why eviction is bottom-up.
• §III.D: cooling-stage sizing (the 10% heuristic) and the hit/second-chance probabilities — Fig. 6 shows random+FIFO tracks LRU closely on Zipf.
• §V: evaluation — in-memory TPC-C at parity with a no-buffer-manager build; the graceful degradation curve as data exceeds RAM (the money plot).

vmcache: the retraction and the fix

Swizzling works but infects the whole codebase: every data structure must know swips, honor one-parent, cooperate with cooling. vmcache keeps the goal and drops the mechanism:

• mmap an anonymous (or exmap) virtual range: page(pid) is just virt + pid * 4096 — the MMU is the translation layer, for free.
• BUT the DB — not the kernel — decides residency: explicit per-page state word (Evicted/Marked/Locked/Unlocked + version counter), explicit reads into the fixed virtual address, madvise(DONTNEED) on evict.
• The page-state word doubles as the hybrid latch (optimistic version).
• Any page can have any number of references — the one-parent rule dies; arbitrary graphs are fine (relevant to a graph-store capstone!).
• The exmap kernel module fixes the syscall/TLB costs of madvise-heavy eviction; without it, plain vmcache still beats classic pools.

The whole design fits in one state machine:

#![allow(unused)]
fn main() {
// Translation is the MMU's job; RESIDENCY is the DB's.
fn page(&self, pid: u64) -> *mut u8 { unsafe { self.virt.add(pid as usize * 4096) } }

fn fix(&self, pid: u64) {
    loop {
        let s = self.state[pid].load();      // Evicted/Marked/Locked/Unlocked + version
        match s.kind() {
            Evicted => if self.state[pid].cas(s, s.locked()) {
                pread(self.fd, self.page(pid), 4096, pid * 4096); // into the FIXED addr
                self.state[pid].store(s.unlocked_bumped());       // word doubles as
                return;                                           // the hybrid latch
            },
            Marked | Unlocked => if self.state[pid].cas(s, s.locked()) { return; },
            Locked => core::hint::spin_loop(),  // someone else is faulting it in
        }
    }
}
// evict: write back if dirty, madvise(DONTNEED, page(pid)), state → Evicted
}

 LeanStore:  translation in POINTERS  (swips, invasive, tree-shaped data)
 vmcache:    translation in the MMU   (virt addressing, any ref-graph)
 both:       replacement + residency decided by the DB, never the kernel

The CIDR ’22 mmap paper (reading-mmap-paper.md) is the missing middle: mmap with kernel-controlled residency is the trap; vmcache is mmap-style addressing with DB-controlled residency.

Questions to answer in notes.md

1. Reproduce LeanStore Fig. 1’s argument as arithmetic: hash probe (topic-0 DRAM numbers) + latch CAS per access, × accesses per TPC-C txn — what fraction of in-memory runtime is the classic pool?
2. Why does the cooling stage need to be a FIFO and not a stack? (Second chance requires time between cool and evict.)
3. vmcache’s page-state word vs postgres’s packed buffer state (buf_internals.h) — same bits, different home. What does colocating state-with-translation (vmcache) buy over a separate descriptor array?
4. For the capstone: GraphBLAS matrix tiles referenced by row and column indexes = a DAG, not a tree. Which of the two designs is even admissible, and what would the swizzling workaround cost?

Done when

You can state what each of the three LeanStore ingredients eliminates, and explain in two sentences why vmcache can drop swizzling without giving back the hot-path win.

References

Papers

• Leis, Haubenschild, Kemper, Neumann — “LeanStore: In-Memory Data Management Beyond Main Memory” (ICDE 2018) — focus on §III.B (one swip per page, bottom-up eviction), §III.D (cooling-stage sizing, Fig. 6), §V (the graceful-degradation money plot)
• Leis, Alhomssi, Ziegler, Loeck, Dietrich — “Virtual-Memory Assisted Buffer Management” (vmcache/exmap, SIGMOD 2023) — read after LeanStore, as the retraction and the fix

LeanStore in code: swips, cooling, hybrid latches

The paper claims a hot page access can cost zero atomics; this chapter walks the classic ICDE ’18 codebase to see how — a u64 that is either a pointer or a page id, a background thread that cools random frames, and latches whose readers hold nothing. Read the paper guide (reading-leanstore-paper.md) first for the why; this is the how.

1. Swip — Swip.hpp:17–67

One u64 that is EITHER a pointer or a page id:

• evicted_bit = 1<<63, cool_bit = 1<<62 (:21–26).
• isHOT() (:45) — both bits clear ⇒ it’s a raw BufferFrame*.
• isCOOL() (:46), isEVICTED() (:47); warm() clears the cool bit (:62), cool() sets it (:65), evict(pid) stores a page id + bit 63 (:67).

The buffer pool’s mapping table is distributed into the parent nodes: no hash lookup, no partition lock, on any hot access. The price: exactly one swip may reference a page (else un/re-swizzling can’t find all pointers).

2. resolveSwip — BufferManager.cpp:281–330

 isHOT   (:283) → return the pointed-to frame. Done. ~0 overhead.
 isCOOL  (:287) → frame exists but sits in the cooling FIFO:
                  latch parent, clear cool bit (second chance), return.
 EVICTED         → page fault: grab free frame, readPageSync (:317),
                  swizzle the swip, return.

The same three arms, as code:

#![allow(unused)]
fn main() {
// The hot path is a pointer dereference — nothing else.
fn resolve(&self, parent: &HybridGuard, swip: &mut Swip) -> &BufferFrame {
    if swip.is_hot() { return swip.frame(); }         // raw pointer: ~0 overhead
    if swip.is_cool() {
        parent.upgrade_exclusive();                   // touched while cooling ⇒
        swip.warm();                                  // second chance: clear the
        return swip.frame();                          // bit, dodge the FIFO
    }
    let frame = self.free_frames.pop();               // EVICTED ⇒ page fault:
    self.read_page_sync(swip.pid(), frame);           // the ONLY case that pays
    swip.swizzle(frame);                              // pid → pointer, in place —
    frame                                             // next access is hot
}
}

Note the latching order comment — BufferManager.hpp:67–68: swizzle vs coolPage acquire latches in conflicting order; the fix is jump-and-retry (optimistic abort) instead of blocking. Deadlock avoidance by restart — the same philosophy as optimistic latches below.

3. The cooling stage — PageProviderThread.cpp

Background thread keeps ~10% of frames “cool”:

• Pick a random buffer frame (:44) — no LRU bookkeeping at all.
• Phase 1 (:52): unswizzle it — but only if all its children are evicted (:90–91, iterateChildrenSwips): evict leaves before parents, bottom-up.
• Cool frames enter a per-partition FIFO (Partition.hpp:65+). Touched while cool ⇒ resolveSwip warms it (cheap save). Reaches FIFO head ⇒ written back if dirty (AsyncWriteBuffer) and evicted.

Random + second-chance approximates LRU with zero per-access cost — compare postgres (per-access usage bump) and DuckDB (per-unpin enqueue). LeanStore pays nothing per access; that’s the whole point of the paper.

4. Hybrid latches — Latch.hpp

• HybridLatch — :26–41: a version word; LATCH_EXCLUSIVE_BIT in the low bit (:41).
• Guard — :51+: OPTIMISTIC state reads the version, proceeds without writing anything, revalidates at the end; version changed ⇒ jump (longjmp -style unwind) and retry. Writers CAS the version odd.
• BufferFrame — BufferFrame.hpp:18–99: latch sits in the header (:27, “NEVER DECREMENT” — versions only grow); isDirty() = page.PLSN != last_written_plsn (:84) — dirtiness derived from LSNs, not a flag. Nice WAL-integration detail for your M6.

This is topic 9’s main subject making an early appearance — for now, note that optimistic readers are what make swizzling safe: a reader holding no pin can’t block eviction, it just fails validation and retries.

Questions to answer in notes.md

1. The one-parent constraint: why exactly does swizzling forbid two swips to the same page? Walk the eviction of a doubly-referenced page. Then decide: do FalkorDB’s tensor/matrix blocks form a tree or a DAG?
2. Bottom-up eviction (children before parents): what breaks top-down? (An evicted parent’s swip can’t hold a hot child’s pointer — the child would be unreachable.)
3. Random candidate selection: estimate hit-rate loss vs true LRU on a Zipf workload (then measure — experiments/benches/eviction.rs has a FIFO arm you can extend with random-cooling).
4. vmcache (SIGMOD ’23) removes swizzling — pages live at virt[pid], the mapping is the MMU’s problem, explicit state machine per page. What of LeanStore survives in it? (Cooling idea stays; swips go; one-parent constraint gone — that’s the headline win.)

Done when

You can draw the swip state machine (HOT/COOL/EVICTED with transitions and who performs each) and explain why a hot hit costs zero atomics.

References

Code

• leanstore/leanstore (the classic ICDE ’18 codebase) — backend/leanstore/storage/buffer-manager/: Swip.hpp, BufferManager.cpp, BufferFrame.hpp, PageProviderThread.cpp, Partition.hpp; latches in backend/leanstore/sync-primitives/Latch.hpp. Local clone at ~/repos/leanstore.

mmap is not a buffer pool

mmap looks like a free buffer pool, and a famous position paper says that for a general-purpose write-heavy DBMS every apparent win reverses. It is short, punchy, and deliberately provocative — so read it adversarially, then construct the counter-evidence yourself (LMDB exists and is excellent). The payoff is knowing precisely which property of a workload makes mmap wrong.

The temptation (§1–2)

mmap looks like a free buffer pool: no copies, no eviction code, pointer access, the kernel’s page cache does the work. Systems that tried: MongoDB (MMAPv1 — abandoned), LMDB (kept it, happily), SQLite (optional), RavenDB… The paper’s claim: for a general-purpose write-heavy DBMS, every apparent win reverses.

The four problems (§3) — memorize these

 1. Transactional safety   kernel may flush a dirty page ANY time
    ────────────────────   ⇒ can't order page-write after log-write
                           ⇒ WAL rule unenforceable without COW tricks
 2. I/O stalls             page fault = your thread stops; no async,
    ────────────────────   no prefetch you control, no admission control
 3. Error handling         disk error = SIGBUS in the middle of a memcpy,
    ────────────────────   not an error code at a syscall boundary
 4. Performance (§4)       the surprise: even READ-ONLY mmap loses at scale

§4 — why read-only mmap still loses (the part worth re-reading)

Three kernel bottlenecks, measured:

• page table contention — single-threaded page-fault handling paths.
• TLB shootdowns — evicting a mapping ⇒ IPI every core that may have the TLB entry: eviction cost scales with core count.
• 4KB granularity + page-table walk overhead vs one big explicit read.

Result (their fio experiment): explicit pread/O_DIRECT sustains device bandwidth; mmap plateaus far below on NVMe arrays and degrades over time once eviction starts.

The rebuttal you must construct (LMDB, topic 3)

LMDB is mmap-based and wins its niche. Why it dodges each bullet: COW pages never overwrite (1: ordering is a non-problem — the meta-page flip IS the commit); read-mostly workloads fault once, then it’s just memory (2); a read-only mmap can’t SIGBUS on writes (3); and its scale target is “fits mostly in RAM” (4). The paper’s own Table 1 concedes designs like this. The honest conclusion: mmap is wrong when the DB must control WRITE-BACK. Read-only/COW designs escape most of it.

Map to what you know

Questions to answer in notes.md

1. Your topic-3 B+tree used explicit I/O. If you’d mmap’d it, which of your topic-5 WAL guarantees break, concretely? (Which test in crash_test.rs would start failing and why.)
2. TLB shootdowns: why does eviction trigger them but faulting-in not?
3. The paper measures read-only workloads losing. Reconcile with LMDB’s read benchmarks winning — what’s different in the setups (working set vs RAM, single NVMe vs array, pointer-chase vs scan)?
4. vmcache’s answer: keep virtual-memory addressing, add explicit state. Which of the four problems does it solve, which does it merely soften?

Done when

You can argue both sides for five minutes each — “never mmap” and “LMDB is right” — and state precisely which property of your workload picks the side.

References

Papers

• Crotty, Leis, Pavlo — “Are You Sure You Want to Use MMAP in Your DBMS?” (CIDR 2022) — short position paper; memorize the four problems of §3, re-read §4 (why even read-only mmap loses at scale)

postgres bufmgr: a buffer’s life in one atomic word

Postgres packs everything CLOCK needs to know about a buffer — refcount, usage count, flags — into a single atomic u64, so the hit path is one CAS and the sweep hand reads victims without locks. This chapter skims the four mechanisms that make the classic shared-buffers design work: the packed state, the hit path, the miss path with foreground dirty-victim flushes, and the background writer that exists to hide them.

1. The packed state — buf_internals.h:49–147

Everything about a buffer fits in ONE atomic u64 (BufferDesc.state, :344):

 ┌──────────── 64-bit state ────────────┐
 │ lock bits │ flags │ usage(4) │ refcount(18) │
 └───────────────────────────────────────┘
 BUF_REFCOUNT_BITS 18 (:49)   BUF_USAGECOUNT_BITS 4 (:50)
 BM_MAX_USAGE_COUNT 5 (:144)  — CLOCK survives ≤5 sweeps

Why packed: pin/unpin/usage-bump are single CAS ops — no spinlock on the hit path. Same trick as topic-2’s SwissTable metadata byte: cram the hot-path-decidable state into one word.

2. The hit path — PinBuffer, bufmgr.c:3295

CAS loop on state: refcount+1, usage_count+1 if < BM_MAX_USAGE_COUNT (:3338–3352). Lookup before that: BufTableLookup under one of NUM_BUFFER_PARTITIONS = 128 partition locks (lwlock.h:83, buf_internals.h:244–250) — the map is sharded so lookups don’t serialize.

Read PinBufferForBlock (:1223) → ReadBuffer_common (:1276) → StartReadBuffersImpl (:1371) for how reads became a vectored/async ReadBuffersOperation (v17+) — the miss path now streams.

3. The miss path — BufferAlloc + GetVictimBuffer

• BufferAlloc — bufmgr.c:2197: lookup (:2224), and on miss call GetVictimBuffer (:2548).
• GetVictimBuffer: StrategyGetBuffer picks a candidate; if it’s dirty, the backend writes it out itself — FlushBuffer right there in the foreground (:2584 onward). Every eviction of a dirty page is a user-visible latency spike; this is what BgBufferSync exists to prevent.
• Note the WAL-rule cameo: XLogNeedsFlush(BufferGetLSN(...)) before a ring buffer flush (~:2633) — can’t write a page whose log isn’t durable.

4. CLOCK — freelist.c

• StrategyControl->nextVictimBuffer — :42, one atomic for the whole pool.
• ClockSweepTick — :104–160: fetch_add(1) % NBuffers, with a CAS-based modular wraparound so the counter never overflows mid-flight.
• StrategyGetBuffer — :184: loop at :246–290 — pinned (refcount ≠ 0) ⇒ skip; usage_count > 0 ⇒ decrement, keep sweeping; both zero ⇒ victim. “no unpinned buffers available” error if everything’s pinned (~:274).

The sweep, distilled:

#![allow(unused)]
fn main() {
// One shared clock hand; a buffer survives ≤5 sweeps untouched.
fn get_victim(&self) -> BufId {
    loop {
        let id = self.clock_tick();                 // fetch_add(1) % NBuffers
        let s = self.desc[id].state.load();
        if s.refcount() != 0 { continue; }          // pinned: invisible to CLOCK
        if s.usage_count() > 0 {                    // recently used: spend a life
            let _ = self.desc[id].state.cas(s, s.dec_usage());
            continue;
        }
        if self.desc[id].state.cas(s, s.pinned()) { // both zero ⇒ victim; pin it
            return id;                              // caller flushes it if dirty —
        }                                           // in the FOREGROUND
    }
}
}

• Buffer rings — GetAccessStrategy :426: a seqscan gets a 256KB private ring (BAS_BULKREAD, :442–459) so one SELECT count(*) on a huge table can’t flush the whole pool. Eviction policy as admission policy.

5. Background writer — BgBufferSync, bufmgr.c:3854

Runs the same clock ahead of the sweep hand, writing dirty buffers so GetVictimBuffer finds clean ones. Pace: bgwriter_lru_maxpages (:190, default 100 pages/round) + a moving average of recent allocation rate (:3877–3911). It’s an estimator — read the long comment.

Questions to answer in notes.md

1. Why 18 bits of refcount but only 4 of usage count? What failure does each cap produce and which is graceful? (usage saturates harmlessly; refcount overflow would be corruption — hence StaticAssert vs MAX_BACKENDS :130.)
2. A client pins a page and crashes mid-query — who unpins? (Resource owner machinery: ReservePrivateRefCountEntry in GetVictimBuffer :2559.)
3. Buffer rings vs LeanStore’s cooling stage: both defend against scans. Which defends at admission and which at eviction? What does each miss?
4. Postgres double-buffers (shared_buffers + OS page cache). What does O_DIRECT (topic 6’s io story, debug_io_direct) buy and cost here?

Done when

You can narrate a miss on a dirty victim end-to-end — every lock, atomic, and I/O in order — and say which step BgBufferSync moves off the hot path.

References

Code

• postgres/postgres — src/backend/storage/buffer/bufmgr.c, src/backend/storage/buffer/freelist.c, src/include/storage/buf_internals.h, src/include/storage/lwlock.h. Local clone at ~/repos/postgres.

zmalloc: memory management when there are no pages

Redis has no buffer pool — no pages, no frames, no eviction hand. What it has instead is an allocation ledger: every malloc accounted on per-thread padded counters, maxmemory enforced against an allocator statistic, and key-level eviction after the fact. This chapter reads that ledger, plus a bonus: turso’s CLOCK page cache in Rust, the closest existing code to your experiment.

1. zmalloc — allocation accounting, not caching

• PREFIX_SIZE — zmalloc.c:39–46: with jemalloc (HAVE_MALLOC_SIZE) the allocator can report a pointer’s size ⇒ prefix is 0; with libc malloc, redis prepends an 8-byte size header to every allocation. The entire used-memory ledger depends on being able to answer “how big is this ptr?”
• used_memory — :86–92: per-thread, cache-line-aligned counters (aligned(CACHE_LINE_SIZE), MAX_THREADS array) — summed on read. False sharing on a global counter would tax every malloc on every thread; same diagnosis as topic 0’s cache-line experiments.
• update_zmalloc_stat_alloc — :105–145: bump my thread’s counter, and only occasionally (peak-check throttle :109–118) pay for the full sum.
• zmalloc — :161–193: malloc_usable_size path vs prefix path.

The ledger, in miniature:

#![allow(unused)]
fn main() {
// One counter per thread, each on its own cache line — a single global
// fetch_add would put coherence traffic on EVERY malloc on EVERY core.
#[repr(align(64))]
struct Padded(AtomicI64);
static USED: [Padded; MAX_THREADS] = /* … */;

fn zmalloc(size: usize) -> *mut u8 {
    let p = unsafe { malloc(size) };
    let real = malloc_usable_size(p);          // jemalloc answers "how big is p?"
    USED[thread_id()].0.fetch_add(real as i64, Relaxed);  // uncontended bump
    p
}
fn used_memory() -> i64 {
    USED.iter().map(|c| c.0.load(Relaxed)).sum()  // the SUM is paid on read,
}                                                 // and reads are rare
}

This ledger is what maxmemory compares against — eviction (LRU/LFU over keys, not pages) triggers on an allocator statistic. The buffer-pool analogue: DuckDB gates allocations up front; redis counts and evicts after.

2. Active defrag — defrag.c

• activeDefragAlloc — defrag.c:177 (+ :142 comment): jemalloc tells redis which allocations sit in sparse bins; redis re-allocates them (new ptr, same bytes) and rewrites every reference. Defragmentation in userspace, cooperatively, because the allocator can’t move memory it handed out.
• FalkorDB angle: GraphBLAS matrices are big opaque zmalloc blobs — redis can count them but not defrag them, and one matrix can blow the maxmemory budget in a single GrB call. Your capstone owns its allocations; decide what “maxmemory” should even mean for a graph store.

3. Bonus: turso’s page cache — ~/repos/turso core/storage/page_cache.rs

A real Rust CLOCK implementation to compare with your experiment after you build it (don’t copy first):

• PageCache — :99–116: intrusive circular list + clock_hand raw pointer (:107); comment :95–98 states the discipline (insert behind the hand).
• advance_clock_hand — :174; insert — :204.
• Note what’s unsafe (Send/Sync impls :115–116, raw pointers) and what your Rust version can do differently with indices into a Vec<Frame> instead of pointers (safe, and the array is exactly postgres’s layout).

Questions to answer in notes.md

1. Why per-thread counters instead of one atomic? Estimate the cost of a shared fetch_add on every malloc at 8 threads (topic-0 numbers).
2. Redis evicts keys; a buffer pool evicts pages. Which gets better hit rates for the same RAM and why is the comparison unfair? (Keys are variable-size and complete — no partial residency of a value.)
3. After building your CLOCK pool: diff your design against turso’s — hand placement on insert, where usage bits live, pin representation.

Done when

You can explain PREFIX_SIZE, why the counters are padded, and what active defrag can’t touch — and you’ve compared your finished pool to turso’s.

References

Code

• redis — src/zmalloc.c (the ledger) and src/defrag.c (cooperative userspace defragmentation). Local clone at ~/repos/redis.
• tursodatabase/turso — core/storage/page_cache.rs, a real Rust CLOCK to diff against your experiment after you build it (don’t copy first). Local clone at ~/repos/turso.

Topic 6 notes — buffer pool & memory management

Predictions (fill BEFORE running)

• pool_vs_mmap p50: mmap ___ ns vs pool ___ ns (who wins the median and why?)
• pool_vs_mmap p99.9: mmap ___ vs pool ___ (who wins the tail?)
• CLOCK hit rate vs strict LRU on Zipf(0.99), 16× universe: gap of ___ %
• strict LRU ns/access vs CLOCK: ___× slower

pool_vs_mmap results

(paste; run warm and note whether the file fit the OS page cache)

Pool hit rate: ___ %. Explanation of the tail difference:

eviction bench results

Verdict — is strict LRU’s hit-rate edge worth its per-hit cost here?

Buffer pool build log (src/buffer_pool.rs)

• Where the WAL rule hooks in (which write, which LSN check):
• What a background writer would take off the miss path:
• What a 6-page scan did to usage counts (hot_page_survives_scan_pressure):

Reading-guide questions

postgres bufmgr (reading-postgres-bufmgr.md)

1. 18 refcount bits vs 4 usage bits — which cap fails gracefully:
2. Who unpins after a crashed query:
3. Buffer rings (admission) vs cooling stage (eviction) — what each misses:
4. What O_DIRECT buys/costs given double buffering:

DuckDB buffer manager (reading-duckdb-buffer.md)

1. Why weak_ptr in eviction nodes:
2. Worst-case dead-node ratio; when CLOCK’s fixed array is strictly better:
3. Throw-on-pressure vs error-when-all-pinned — capstone choice:

LeanStore (reading-leanstore.md)

1. One-parent constraint walkthrough; are FalkorDB matrix blocks a tree or DAG:
2. Why eviction must be bottom-up:
3. Random-cooling hit-rate loss vs LRU (estimate, then measure):
4. What survives in vmcache, what dies:

mmap paper (reading-mmap-paper.md)

1. Which topic-5 crash_test would fail under mmap and why:
2. Why eviction (not fault-in) triggers TLB shootdowns:
3. Reconciling the paper with LMDB’s wins:
4. vmcache vs the four problems — solved vs softened:

LeanStore/vmcache papers (reading-leanstore-paper.md)

1. Classic-pool overhead as arithmetic (topic-0 numbers):
2. Why cooling is a FIFO, not a stack:
3. vmcache state word vs postgres packed state — what colocation buys:
4. Which design is admissible for matrix-tile DAGs:

redis zmalloc (reading-redis-zmalloc.md)

1. Cost of a shared fetch_add per malloc at 8 threads:
2. Key eviction vs page eviction — why the comparison is unfair:
3. My pool vs turso’s page_cache.rs — three design diffs:

M6 log (capstone)

• Buffer pool under the persistent backends
• Per-backend pools vs shared pool + MemoryTag accounting — decision + why:
• mmap write-back unpredictability reproduced (numbers):

Topic 7 — Networking, Protocols & Event Loops

Redis’s speed is as much about ae.c and RESP as about data structures. You know FalkorDB’s module side — this topic is about owning the server side: one loop, many sockets, and a protocol designed to be parsed with memchr.

Outcomes

By the end you can:

1. Parse and generate RESP2/RESP3 from memory, and explain each design choice (length prefixes, CRLF, type-first bytes) in parser terms.
2. Narrate one full redis event-loop iteration: poll → read → parse → execute → queue reply → write, and say what beforeSleep does.
3. Explain the three threading models (single loop, io-threads, thread-per- core) and what each serializes.
4. Ship a tokio RESP server that survives redis-benchmark, and read its flamegraph.

1. RESP: a protocol optimized for the parser

 client:  *2\r\n $3\r\n GET\r\n $3\r\n foo\r\n
          │      │                └ bulk string, length-prefixed: read(3)
          │      └ each arg: $<len> — NO scanning for terminators
          └ array header: argc up front — allocate argv once

 server:  $3\r\n bar\r\n         +OK\r\n        :42\r\n      -ERR msg\r\n
          bulk                   simple          integer      error

Why it’s fast to parse:

• Length prefixes everywhere — the parser never scans payload bytes; it reads a small integer, then memcpys exactly that many. Binary-safe for free. (Contrast: HTTP/1 header parsing scans for \r\n\r\n.)
• argc first — *N lets the server size argv[] before reading args.
• First byte = type — a one-byte dispatch, no lookahead.
• RESP3 adds typed replies (maps %, sets ~, doubles ,, push >) so clients stop guessing structure from context — same wire discipline.

Inline commands (PING\r\n) exist purely so you can debug with nc.

2. The event loop

flowchart TB
    A["aeMain: while !stop"] --> B["beforeSleep():<br/>flush AOF, handle<br/>PENDING WRITES first"]
    B --> C["aeApiPoll (kqueue/epoll)<br/>wait for readable/writable"]
    C --> D["for each ready fd:<br/>readQueryFromClient"]
    D --> E["parse RESP →<br/>execute command →<br/>addReply to OUTPUT BUFFER"]
    E --> B

The two non-obvious moves:

• Replies are buffered, not written — addReply appends to a per-client buffer and the next beforeSleep writes everything with one write() per client (handleClientsWithPendingWrites). Batching by loop iteration.
• Pipelining falls out for free — the input buffer may hold 100 commands; processInputBuffer loops until the buffer is drained, and all 100 replies coalesce into one write. This is why redis-benchmark -P 64 is ~10× -P 1: same work, 1/64th the syscalls.

3. Three threading models, one question: what’s serialized?

io-threads keep redis’s contract (commands are atomic, no locks in data structures) and parallelize only the syscall+parse layer — valkey 8’s rework made the main thread hand batches over SPSC queues and prefetch dict entries before executing (memory stalls, topic-0 style, hidden by batching). Thread-per-core abandons the shared keyspace instead: hash-partition keys to cores, cross-slot ops become messages. FalkorDB inherits redis’s model: one graph = one keyspace entry ⇒ module-level locking is the concurrency story.

4. Backpressure — the part everyone forgets

• Input side: PROTO_IOBUF_LEN 16KB reads (server.h:188), max query buffer size; a client streaming faster than execution grows querybuf → kill.
• Output side: PROTO_REPLY_CHUNK_BYTES 16KB chunks (server.h:189); a slow reader (or a KEYS * on 10M keys) grows the reply list → client-output-buffer-limit → kill. A database is a flow-control device: a graph query returning 1M rows must either stream with backpressure (pgwire’s row-at-a-time) or buffer-and-die (redis’s approach).
• pgwire contrast: postgres streams DataRow messages inside a Simple/Extended Query dance with per-portal row limits — the protocol itself has backpressure hooks RESP lacks.

5. Bolt: the third answer (RESP vs pgwire vs Bolt)

Three protocols, three answers to the same three questions:

Bolt is what a protocol looks like when the data model lives in the protocol: PackStream has markers for maps, lists — and graph types (Node 0x4E, Relationship 0x52, Path 0x50), so a driver hands you a graph object, not a string table. And streaming is client-driven: after RUN, records flow only when the client asks (PULL {n:1000}) — backpressure designed in, not bolted on (section 4’s problem, solved at the protocol layer). Versioned handshake: 4 bytes magic 0x6060B017 + four proposed versions; the server picks. → reading-bolt-packstream.md — Bolt & PackStream: the graph in the type system

6. Code reading (5–7 h)

• redis ae.c + networking.c — the loop, the parse path, pending writes. → reading-redis-ae-networking.md — The redis event loop: pipelining for free
• valkey io-threads rework — SPSC job queues, command-batch prefetch. → reading-valkey-iothreads.md — valkey io-threads: parallelize the majority, nothing else
• pgwire (Rust) + qdrant’s tonic setup — what a protocol crate looks like; gRPC as the anti-RESP. → reading-pgwire-qdrant.md — pgwire & tonic: sessions, portals, and protocols you don’t write
• FalkorDB’s removed Bolt server — git show 0b11a00b3^:src/bolt/ (it was deleted in #2170; the tree one commit back is a complete, compact Bolt 5.x implementation). → reading-bolt-packstream.md — Bolt & PackStream: the graph in the type system

7. Reading (2–3 h)

• “The C10K problem” (Kegel) — the historical why of event loops. → reading-c10k-thread-per-core.md — C10K to thread-per-core: what is a server thread for? (covers Glauber Costa’s thread-per-core essays + valkey multithreading blog posts in the same guide)

8. Experiments (in experiments/)

1. src/resp.rs — RESP2 parser/encoder (your build; tests fix the format incl. partial-input resumption — the hard part of any wire parser).
2. src/bin/server.rs — tokio GET/SET/PING/DEL server over your parser
   • a sharded HashMap. Compiles against your resp.rs.
3. Bench protocol (in notes.md): redis-benchmark -t get,set -P 1 and -P 64 against (a) your server, (b) real redis on this Mac. Flamegraph your server under load; name the top 3 entries.

9. Capstone milestone M7 (in ../../capstone/)

• RESP server exposing GRAPH.QUERY / GRAPH.RO_QUERY, wire-compatible with falkordb-py (the client must not know it’s not FalkorDB).
• Bench falkordb-py against yours vs real FalkorDB; document the gap.
• Decide + write down: single loop, io-threads, or thread-per-core — and what your choice serializes.
• Stretch: a Bolt listener on a second port so neo4j drivers connect — PackStream encoding of the graph result types (Node/Relationship/Path).

Done when

Your server handles redis-benchmark -P 64 without protocol errors; you can write *3\r\n$3\r\nSET\r\n… from memory; you can explain why pipelining multiplies throughput without touching command execution.

Bolt & PackStream: the graph in the type system

RESP encodes a node as nested arrays the client must re-interpret; Bolt puts Node, Relationship, and Path on the wire as first-class types, and makes result streaming client-driven — backpressure IS the protocol. The reference implementation here is FalkorDB’s own Bolt 5.x server, complete until #2170 removed it (2026-07-08): read it frozen in time with git show 0b11a00b3^:src/bolt/<file> in ~/repos/FalkorDB.

The shape of a Bolt session

client                                server
  │ 0x60 0x60 0xB0 0x17 + 4 versions   │  handshake: bolt_api.c:803,
  ├────────────────────────────────────►│  version pick :845-864
  │◄──────────────── chosen version ────┤  (5.1..5.7 accepted)
  │ HELLO {auth...}          0x01       │
  │◄─────────────── SUCCESS  0x70 ──────┤
  │ RUN "MATCH..." {} {}     0x10       │  bolt_api.c:721
  │◄─────────────── SUCCESS {fields} ───┤  (query ran; no rows sent!)
  │ PULL {n: 1000}           0x3F       │  bolt_api.c:726
  │◄─────────────── RECORD × n  0x71 ───┤  client-driven streaming:
  │◄─────────────── SUCCESS {has_more}──┤  backpressure IS the protocol
  │ DISCARD                  0x2F       │  (or: stop paying for rows)

Every message and value is one PackStream structure: a marker byte 0xB0+size (bolt.c:36), a tag byte (the BST_* enum, bolt.h:27), then fields. The whole message vocabulary — HELLO/RUN/PULL/DISCARD, and the data types Node 0x4E / Relationship 0x52 / Path 0x50 — lives in one enum. RESP has no equivalent: a FalkorDB RESP reply encodes a node as nested arrays the client library must re-interpret; Bolt puts the graph in the type system.

PackStream in one sitting (bolt.c)

• Markers: NULL 0xC0 (bolt.c:11), tiny-string base 0x80 (:21), structure base 0xB0 (:36) — high nibble = type, low nibble = size for “tiny” variants.
• bolt_reply_int (bolt.c:133) picks tiny-int/int8/16/32/64 by value — varint-by-cases, biased so -16..127 costs one byte.
• bolt_reply_structure (:250), lists (:198), maps (:225): everything nests; a Path is a structure of lists of Node/Relationship structures.
• Compare topic 7 §1: RESP optimizes the parser (scan for \r\n); PackStream optimizes the type round-trip (marker dispatch table).

The marker scheme, as an encoder:

#![allow(unused)]
fn main() {
// High nibble = type, low nibble = size for "tiny" variants; ints are
// varint-by-cases, biased so -16..127 costs exactly one byte.
fn write_int(out: &mut Vec<u8>, v: i64) {
    match v {
        -16..=127 => out.push(v as u8),                                        // tiny
        _ if i8::try_from(v).is_ok()  => { out.push(0xC8); out.push(v as u8); }
        _ if i16::try_from(v).is_ok() => { out.push(0xC9); out.extend((v as i16).to_be_bytes()); }
        _ if i32::try_from(v).is_ok() => { out.push(0xCA); out.extend((v as i32).to_be_bytes()); }
        _ => { out.push(0xCB); out.extend(v.to_be_bytes()); }
    }
}
fn write_struct_header(out: &mut Vec<u8>, n_fields: u8, tag: u8) {
    out.push(0xB0 + n_fields);   // marker: tiny structure of n fields
    out.push(tag);               // 0x4E Node, 0x52 Relationship, 0x50 Path, 0x10 RUN…
}                                // then the fields follow, each PackStream-encoded
}

Server-side mechanics worth stealing

• BoltRequestHandler (bolt_api.c:670): one dispatch switch over BST_* — the protocol state machine is ~10 cases.
• RUN executes the query but replies only metadata (:467-482); records flow in the PULL handler (:504-521) — result materialization and result transport are decoupled server-side too.
• ws_handshake (bolt_api.c:831): the same port sniffs and upgrades WebSocket — that’s how browser clients speak Bolt.
• It’s all inside a Redis module: the Bolt socket bypasses RESP entirely and injects work into the same executor — two protocols, one engine.

Questions

1. RUN/PULL splits “execute” from “fetch”. What does the server have to hold between the two, and what does that cost under 10K idle cursors? (Compare pgwire portals, topic 7 §4.)
2. PackStream has no length prefix on messages — chunking (2-byte chunk headers, 0x0000 terminator) wraps it. Why chunk instead of length-prefixing the whole message, for a server that streams records as it produces them?
3. The handshake proposes four versions, server picks one (bolt_api.c:845-864 clamps to 5.1..5.7). Compare RESP’s HELLO 2/3. Which design lets a proxy transparently downgrade, and why?
4. Node/Relationship on the wire carry element ids + property maps. What does this rule out that RESP’s “everything is arrays” allows — and which side of the trade does a new graph database want?
5. Why might FalkorDB have removed Bolt (#2170)? List the real costs a second protocol imposes on an engine (state machines, result encoders, auth, tests, fuzz surface) — then what you’d need to keep it cheap.
6. M7 mapping: the stretch goal is a Bolt listener beside RESP. Which pieces of your M7 RESP server are protocol-neutral (executor, result set) and which need a Bolt twin? Sketch the bolt_reply_*-equivalent trait your result set must implement.

References

Papers

• Neo4j — Bolt Protocol + PackStream specifications (https://neo4j.com/docs/bolt/current/) — the normative source for markers, messages, and the handshake

Code

• FalkorDB/FalkorDB src/bolt/ (bolt.c, bolt.h, bolt_api.c) — removed by #2170; read it frozen in time with git show 0b11a00b3^:src/bolt/<file> in ~/repos/FalkorDB

C10K to thread-per-core: what is a server thread for?

Three readings spanning 1999→2024, one thread: what should a server thread be responsible for? Kegel’s C10K catalog explains why event loops became the default, valkey’s 8.0 posts show disciplined Amdahl analysis parallelizing exactly the profiled majority, and Glauber Costa’s thread-per-core essays take the radical endpoint — share nothing between cores. Together they span the shared↔sharded plane M7 must position itself in.

1. “The C10K problem” — Dan Kegel (kegel.com/c10k.html)

Read it as history that explains present defaults. 1999’s question: how do you serve 10,000 sockets when a thread per connection costs a stack + a scheduler slot each?

The menu it catalogs (skim the I/O-strategy section, skip the dated driver patches):

1. thread per connection — dies at ~10K in 1999 (stacks, context switches)
2. select/poll — O(n) scans of the fd set per wakeup
3. readiness notification (epoll/kqueue) — O(ready) not O(registered): this line wins; ae.c is exactly this + a dispatch table
4. async I/O (POSIX aio) — stillborn on Linux for sockets; the idea returns as io_uring twenty years later

What changed since: threads got cheaper (10K threads is fine now), cores multiplied (one loop can’t fill a 64-core box), and NICs got faster than a single core’s syscall budget. Hence the two modern answers below.

2. Valkey’s multithreading blog posts (valkey.io/blog — the 8.0

performance series)

Read after reading-valkey-iothreads.md; the posts give the measured story:

• redis-6-style io-threads: threads spin-wait, main thread coordinates every batch — modest gains, high CPU burn.
• valkey 8 rework: SPSC handoff, threads own the whole read→parse and write path, main thread prefetches for batches ⇒ ~1M+ ops/s/node claims, ~2–3× over redis 7 on the same box.
• The reasoning to internalize: they profiled first — parse+syscall was the majority of CPU at high op rates; commands themselves were ~30%. The rework parallelizes exactly the majority and nothing else. (Amdahl, applied with discipline.)

3. Glauber Costa on thread-per-core (the Seastar/ScyllaDB and later

Glommio essays: “The reactor pattern is dead, long live the reactor”, shard-per-core posts)

The radical position: don’t share ANYTHING between cores.

• One reactor per core, connections pinned, data sharded by core — a request for shard 7 arriving on core 2 is forwarded as a message (cross-core SPSC again), never locked.
• No locks ⇒ no lock contention, but also: no work stealing ⇒ a hot shard is a hot core; tail latency now depends on your partitioning function.
• Rust incarnation: Glommio (io_uring + thread-per-core executors) vs tokio’s work-stealing multi-thread runtime. Tokio moves tasks to idle workers (great for uneven load, pays cross-core cache traffic); Glommio never moves them (great cache locality, pays imbalance).

        shared keyspace ◄──────────────────────► sharded keyspace
 redis/valkey: 1 exec thread     DragonflyDB/Scylla: N exec threads,
 + io threads, zero data locks   keyspace hash-partitioned per core,
                                 cross-shard ops = messages/transactions

Questions to answer in notes.md

1. Which C10K strategy is tokio’s multi-thread runtime? (Careful: it’s readiness-based mio underneath + work-stealing tasks on top — two layers, name both.)
2. A graph database’s unit of work is a query (ms-scale), not a GET (µs-scale). Redo valkey’s Amdahl analysis for M7: what fraction of a GRAPH.QUERY round-trip is parse+I/O, and does ANY threading of the network layer matter? Where do the threads belong instead (M9)?
3. Thread-per-core for a graph: matrices don’t hash-partition like a keyspace. What’s the sharding unit — graph? subgraph? matrix tile? What does a BFS crossing shards cost in messages?
4. io_uring (the C10K “async I/O” line resurrected): what changes in ae.c’s design if poll+read+write become submission-queue entries? (Topic 6’s O_DIRECT thread rejoins here.)

Done when

You can place redis, valkey 8, tokio, and DragonflyDB in the shared↔sharded / loop↔threads plane and argue M7’s position in it.

References

Papers

• Dan Kegel — “The C10K problem” (kegel.com/c10k.html, 1999–2003) — skim the I/O-strategy section, skip the dated driver patches
• Valkey blog — the 8.0 performance/multithreading series (valkey.io/blog) — read after reading-valkey-iothreads.md for the measured story
• Glauber Costa — thread-per-core essays (“The reactor pattern is dead, long live the reactor”; the Seastar/ScyllaDB shard-per-core posts and later Glommio writing)

pgwire & tonic: sessions, portals, and protocols you don’t write

Two contrasts with RESP: a protocol with stateful sessions and streaming (postgres wire, via the pgwire Rust crate), and a protocol you don’t write at all (gRPC, via qdrant’s tonic setup). Together they bracket RESP’s design point — no handshake, no cursors, buffer-or-die — and fill in the design-space table M7 has to take a position on.

1. pgwire — ~/repos/pgwire

The crate structure IS the protocol lesson:

• src/messages/ — every frontend/backend message as a typed struct with encode/decode. Postgres framing: 1 type byte + i32 length + payload — like RESP’s type-first byte but with binary length (RESP: ASCII digits).
• src/api/query.rs — the two query protocols:
  • SimpleQueryHandler — :48: one Query message in, a stream of RowDescription + DataRow* + CommandComplete out. RESP-like.
  • ExtendedQueryHandler — :174: Parse → Bind → Execute → Sync, five round-trips of state. Prepared statements, parameter binding, binary result formats, and portals — a suspended query you pull N rows from. This is protocol-level backpressure and cursoring; RESP has neither (a module either buffers the whole reply or blocks the loop).
• src/api/auth.rs — StartupHandler (see api/mod.rs:555): the connection is a state machine from byte 0 — startup params, auth exchange, then ready-for-query. RESP connections have no handshake at all (HELLO is optional) — count what that costs postgres in connection setup and what it buys (per-session GUCs, tx state, cancel keys).

Read it asking: where does session state live? — pgwire forces a ClientInfo through every call; your RESP server keeps per-connection state implicitly in the task. Both are answers to “protocol = state machine”.

The extended-query state machine, distilled:

#![allow(unused)]
fn main() {
// The protocol IS a session state machine; portals are protocol-level
// backpressure — a suspended query the client pulls N rows at a time.
match msg {
    Parse { name, sql }          => { self.stmts.insert(name, prepare(sql)?); }
    Bind { portal, stmt, args }  => { self.portals.insert(portal, cursor(stmt, args)?); }
    Execute { portal, max_rows } => {
        let cur = self.portals.get_mut(&portal)?;
        for row in cur.take(max_rows) { send(DataRow(row))?; }
        if cur.done() { send(CommandComplete)?; }
        else          { send(PortalSuspended)?; }   // client decides when to pull more
    }
    Sync => { self.close_txn_if_failed(); send(ReadyForQuery)?; }
    _ => { /* Describe, Close, Flush … */ }
}
}

2. qdrant — ~/repos/qdrant/src/tonic/

• src/tonic/mod.rs:138 and :277 — Server::builder() twice: separate internal (peer-to-peer raft) and public gRPC servers. Protocol surface split by trust domain — compare redis exposing admin + data on one port.
• The services are generated from .proto (see api/ crate: qdrant’s protos) — the parser, framing, streaming, and backpressure (HTTP/2 flow control windows) are inherited, not written. The cost: every message is protobuf — field tags, varints, no zero-copy into your value types; and HTTP/2 framing means you can’t debug with nc.
• Note the middleware layers in mod.rs (auth :138 area, logging, telemetry) — tower’s onion model vs redis’s “check ACL inside processCommand”.

3. The design-space table (fill the last row yourself)

Questions to answer in notes.md

1. FalkorDB result sets ride RESP arrays — huge ones buffer entirely in the module. What would a portal-style GRAPH.QUERY cursor look like as RESP commands? (FalkorDB actually has one — recall GRAPH.QUERY’s timeout + result-set config; design the missing GRAPH.CURSOR anyway.)
2. Why does qdrant run TWO tonic servers instead of one with authz? What attack/ops story does the split simplify?
3. Extended query’s 5 messages cost a round-trip each unless pipelined — how does pgwire’s async design let Parse/Bind/Execute/Sync coalesce, and what’s the RESP equivalent? (MULTI? No — pipelining itself.)

Done when

You can fill the table’s last row with committed answers for M7 and defend “RESP + explicit cursor commands” against “just use gRPC” for a graph DB.

References

Code

• sunng87/pgwire — src/messages/, src/api/query.rs, src/api/auth.rs; the crate structure IS the protocol lesson. Local clone at ~/repos/pgwire.
• qdrant/qdrant — src/tonic/mod.rs (two servers, tower middleware) plus the generated services in the api/ crate. Local clone at ~/repos/qdrant.

The redis event loop: pipelining for free

One thread, one poll syscall per iteration, and two buffering decisions — parse everything the read buffer holds, write nothing until beforeSleep — give redis pipelining and reply batching without any dedicated machinery. ae.c is ~500 lines, so read it fully (a rare luxury); networking.c is huge, so read only the five functions this chapter walks.

1. ae.c — the whole loop

• aeCreateEventLoop — ae.c:47: arrays indexed by fd (events[fd]), not a hash — fds are small dense integers, the OS hands you the perfect array index. setsize = maxclients + headroom.
• aeProcessEvents — :360: the core. beforesleep callback (:377–378), then aeApiPoll (:398) — ONE syscall per iteration collects all ready fds.
• Backend selection: ae_kqueue.c (your Mac), ae_epoll.c (Linux), compile-time — the abstraction is 4 functions (add/del/poll/name).
• Timers ride the same loop: poll timeout = time to nearest timer.

2. The read path — networking.c

• readQueryFromClient — :3715: connection is readable ⇒ read up to 16KB (PROTO_IOBUF_LEN, server.h:188) into querybuf, then parse.
• processInputBuffer — :3529: loop — parse as many complete commands as the buffer holds, executing each. This loop IS pipelining: 100 commands in one read = 100 executions, zero extra syscalls.
• processMultibulkBuffer — :3117: the RESP parser. Read *argc (:3123– 3157), then per arg read $len then exactly len bytes. Note PROTO_MBULK_BIG_ARG (server.h:191, 32KB): big args get the querybuf repositioned so the arg can become an sds object without a copy — zero-copy for large SETs.
• processInlineBuffer — :2968: the nc-friendly fallback — scan for newline (:2975), split on spaces. The ONLY scanning parser in the path.
• Incomplete input ⇒ return, keep bytes in querybuf, wait for the next readable event. State lives in multibulklen/bulklen (:184–185) — your Rust parser’s resumption test mirrors exactly this.

The read path’s shape, in one loop:

#![allow(unused)]
fn main() {
// processInputBuffer: drain every COMPLETE command the buffer holds.
// This loop IS pipelining: 100 commands in one read() = 100 executions,
// zero extra syscalls.
fn process_input(&mut self, c: &mut Client) {
    loop {
        match parse_multibulk(&c.querybuf[c.pos..]) {  // *argc, then $len + bytes per arg
            Parsed { cmd, consumed } => {
                c.pos += consumed;
                execute(&cmd, c);                      // addReply BUFFERS, never writes
            }
            Incomplete => break,      // keep the bytes; multibulklen/bulklen remember
        }                             // where we were — resume on the next readable event
    }
    c.querybuf.drain(..c.pos);
    c.pos = 0;
}
}

3. The write path — the part that surprises people

• addReply — :572: does NOT write to the socket. Appends to the client’s reply buffer/list and flags the client as pending-write.
• handleClientsWithPendingWrites — :2802: called from beforeSleep — walk pending clients, writeToClient each (one write() syscall for ALL replies accumulated this iteration). Socket buffer full ⇒ install a write handler and let the loop wake us when writable (the only time redis uses write events).
• Reply buffering structure: fixed 16KB buffer first (PROTO_REPLY_CHUNK_ BYTES), overflow into a list of blocks — small replies never allocate.

4. Backpressure

Find closeClientOnOutputBufferLimitReached (grep it): a slow consumer or a huge reply grows the list until the configured limit kills the client. Trace what happens when GRAPH.QUERY returns 1M rows through a module: module → RedisModule_ReplyWith* → these same buffers → possibly the axe.

Questions to answer in notes.md

1. Why write in beforeSleep rather than in addReply? Count syscalls for a pipeline of 100 GETs both ways.
2. events[fd] arrays vs a HashMap<fd, handler>: why is the array not just faster but correct here? (fd reuse semantics after close.)
3. The big-arg zero-copy: what property of sds + querybuf repositioning makes it safe? When does it fail (arg spans two reads)?
4. Your tokio server does a write per response future by default — what’s the tokio equivalent of pending-writes batching? (Hint: buffered writer + flush on yield, or explicit corking.)

Done when

You can narrate one loop iteration with 3 pipelined clients — every syscall, every buffer — and explain where a 101st slow client changes the story.

References

Code

• redis — src/ae.c (read fully), src/networking.c (the five functions above), plus the buffer-size constants in src/server.h. Local clone at ~/repos/redis.

valkey io-threads: parallelize the majority, nothing else

Valkey 8 rewrote redis 6’s io-threads and roughly doubled throughput — while commands still execute on one thread with zero locks in the data structures. Read it as a case study in what to parallelize when you refuse to lock the data structures: SPSC handoff, batch commit, and a prefetcher that turns pointer chases into a pipeline. This is the “great perf PRs to study” item.

0. The contract

Commands still execute ONLY on the main thread — single-threaded semantics, zero locks in dict/rax/etc. What moves to threads: read(), RESP parsing, write(), and (new) memory prefetching. Amdahl says: that’s worth it exactly when parse+I/O dominates — i.e., small commands, many clients. GRAPH.QUERY with 50ms of matrix math? io-threads buy ~nothing. GET/SET at 1M ops/s? 2×.

1. The plumbing — io_threads.c

• IOThreadMain — :293: each thread’s loop. Priority 1: drain its private SPSC queue in batches of BATCH_SIZE (:320–321, spscDequeueBatch), jobs are tagged pointers (untagJob :333 — job type in the pointer’s low bits, topic-2’s bit-smuggling again).
• io_private_inbox[IO_THREADS_MAX_NUM] — :23: one SPSC queue per thread. Single-producer (main thread), single-consumer (that io thread) ⇒ no CAS contention at all; compare redis 6’s design where threads spun on a shared list with a busy-wait fence.
• spscCommit (:61) — the producer batches enqueues and commits once: amortizing the release-store, same group-commit shape as topic 5.
• trySendReadToIOThreads — :514 / trySendWriteToIOThreads — :550: main thread offloads a client if threads are enabled + client is eligible; networking.c calls these at :2313, :3043, :6408 — note every call site has a same-thread fallback.
• initIOThreads — :489; threads can be resized at runtime (:476).

2. The clever part — memory_prefetch.c

Commands parsed by io threads sit in a batch; before the main thread executes them, it prefetches the dict entries the batch will touch:

• PrefetchCommandsBatch — :26–33: keys of up to max_prefetch_size commands from multiple clients.
• The file comment (:7) states the idea: walk each key’s lookup path (hash → bucket → entry → value) issuing __builtin_prefetch at each level, interleaved across keys — while key A’s bucket line is in flight, compute key B’s hash. This is software memory-level parallelism: topic 0’s MLP finding (hash lookups are flat because loads overlap) engineered deliberately.

 without: exec(A): miss…wait 100ns… exec(B): miss…wait…      serial misses
 with:    prefetch A.bucket, B.bucket, C.bucket  (overlap!)
          exec(A) hit, exec(B) hit, exec(C) hit               misses paid once

The interleaving, made explicit:

#![allow(unused)]
fn main() {
// Walk every key's lookup path LEVEL BY LEVEL across the batch:
// while A's bucket line is in flight, compute B's hash — the pointer
// chase becomes a pipeline of overlapping misses, not a chain.
fn prefetch_batch(dict: &Dict, batch: &[Command]) {
    let hashes: Vec<u64> = batch.iter().map(|c| hash(c.key())).collect();
    for &h in &hashes {
        prefetch(dict.bucket_addr(h));          // level 1: all bucket lines
    }
    for &h in &hashes {
        prefetch(dict.entry_addr(h));           // level 2: entries (buckets now warm)
    }
    // main thread then executes the batch: every lookup hits warm lines
}
}

3. What to steal for M7

• SPSC per worker beats MPMC when you can dedicate pairs — in tokio terms: per-connection tasks already give you this shape for free; the lesson applies when you add a worker pool for query execution (M9).
• Batch handoff + commit, not per-item signaling.
• Prefetch only helps when execution is memory-bound on predictable pointer chains — matrix kernels are already streaming; the graph-store analogue is prefetching node/edge attribute blocks for a batch of lookups.

Questions to answer in notes.md

1. Why SPSC queues instead of one MPMC queue? What does the redis-6 design (shared list + busy spin) pay per job that SPSC doesn’t?
2. Tagged job pointers: why smuggle the type in low bits instead of a struct { type, ptr }? (Queue slot stays one word ⇒ one cache line moves per batch of 8.)
3. Amdahl accounting for FalkorDB: measure (or estimate) parse+I/O share of a GRAPH.QUERY round-trip; at what query cost does io-threading stop mattering?
4. Why must prefetch batches span multiple clients to work? (One client’s pipeline is sequential in the buffer, but its keys are independent — what actually limits batch depth?)

Done when

You can explain what valkey parallelized, what it deliberately didn’t, and why the prefetcher is the same insight as topic 0’s MLP experiment.

References

Code

• valkey-io/valkey — src/io_threads.c, src/memory_prefetch.c (the file comment at :7 states the whole idea), plus the grep points in src/networking.c. Local clone at ~/repos/valkey.

Topic 7 notes — networking, protocols & event loops

Predict FIRST, then measure. Numbers without predictions are just trivia.

Predictions (fill in BEFORE running anything)

Reasoning space (why did you predict those numbers?):

• At -P 1 the bottleneck is syscalls + RTT, not parsing — both servers should be within ~2× of each other.
• At -P 64 parse+execute per syscall dominates; where does the re-parse-from- buffer-start simplification in resp.rs cost show up, if anywhere?

Bench protocol

# your server
cd experiments && cargo run --release --bin server
redis-benchmark -p 7379 -t get,set -n 1000000 -P 1
redis-benchmark -p 7379 -t get,set -n 1000000 -P 64

# real redis (brew services or redis-server), port 6379
redis-benchmark -p 6379 -t get,set -n 1000000 -P 1
redis-benchmark -p 6379 -t get,set -n 1000000 -P 64

# flamegraph (cargo install flamegraph; needs sudo for dtrace on macOS)
cargo flamegraph --release --bin server
# then drive load with redis-benchmark -P 64 from another terminal

Record: req/s for each cell, and the top-3 flamegraph entries with rough %.

Questions — reading-redis-ae-networking.md

1. Why does the RESP parser never scan payload bytes? What property of the wire format makes that possible, and what does the inline protocol lose?
2. Trace one GET at the function level: aeApiPoll → readQueryFromClient → processInputBuffer → processMultibulkBuffer → addReply → handleClientsWithPendingWrites. Where is the syscall boundary crossed (exactly twice — where)?
3. What do multibulklen/bulklen (server.h:184–185) buy over re-parsing from the buffer start (what your resp.rs does)? Estimate the cost difference for a 1MB bulk arriving in 16KB reads.
4. PROTO_MBULK_BIG_ARG (:191): what copy does the zero-copy path avoid, and why does it only matter above 32KB?

Questions — reading-valkey-iothreads.md

1. What exactly does an io thread own in valkey 8 (read side, write side), and what remains main-thread-only? Why is that split Amdahl-optimal for GET-shaped workloads?
2. How does the tagged-pointer inbox (untagJob :333) avoid a second queue — and where have you seen bit-smuggling like this before (topics 0, 6)?
3. memory_prefetch.c batches dict lookups to overlap cache misses. What is the equivalent MLP opportunity in a GraphBLAS BFS step?
4. Redo the Amdahl analysis for FalkorDB: GRAPH.QUERY spends how much in parse+I/O vs execution? Does io-threading the network layer move the needle at all for ms-scale queries?

Questions — reading-pgwire-qdrant.md

1. Extended query protocol: what do Parse/Bind/Execute/Sync + portal max_rows give the client that RESP fundamentally cannot? (Hint: who controls the flow of a huge result set?)
2. RESP kills clients that exceed output-buffer limits; pgwire suspends portals. Which does FalkorDB inherit, and what does that mean for a query returning 10M nodes?
3. Why does qdrant run two tonic servers instead of one with auth middleware? When would M7 want the same split?

Questions — reading-c10k-thread-per-core.md

1. Which C10K strategy is tokio’s multi-thread runtime? (Two layers — name both.)
2. A graph database’s unit of work is a query (ms), not a GET (µs). Where do threads belong: network layer (M7) or executor (M9)? Argue with the valkey numbers.
3. Thread-per-core sharding unit for a graph: graph? subgraph? matrix tile? What does a BFS crossing shards cost in messages?
4. io_uring: what changes in ae.c’s design if poll+read+write become submission-queue entries?

Design decisions (record as you implement resp.rs)

• Incomplete-input handling: peek-then-commit or re-parse from start? What state would you keep to make it O(new bytes) instead of O(buffered)?
• Where does Value allocate? Count allocations for *2 $3 GET $3 foo — redis does this with zero per-arg heap allocations at steady state (sds reuse); how many do you do?
• Error strategy: why kill the connection on protocol error instead of resyncing? (What would resync even mean without a framing marker?)

Threading-model placement (do after all readings)

Place on the shared↔sharded / loop↔threads plane:

                 single loop          io threads           thread-per-core
shared keyspace  redis ≤6             valkey 8              (contention!)
sharded keyspace   —                    —                  DragonflyDB/Scylla

Where does M7 sit, and why? (Consider: matrices don’t hash-partition, queries are ms-scale, GraphBLAS wants big parallel sections in the EXECUTOR not the network layer.)

M7 log (capstone milestone)

• resp.rs passes all 8 tests
• server survives redis-benchmark -P 64 for 1M ops
• benched vs real redis, both pipelining levels, numbers above
• flamegraph captured; top-3 named and explained
• GRAPH.QUERY wire-compat: falkordb-py client can connect to the M7 server and get a well-formed (stub) reply
• threading-model decision written down with the Amdahl argument

Topic 8 — Transactions & MVCC

The intellectual core of OLTP. Everything here is one question asked five ways: when two transactions touch the same data, who sees what, and who must die?

Budget: ~12 h. Order: §1 anomalies → §2 three concurrency schools → §3 postgres on-disk MVCC → §4 in-memory MVCC → experiments → M8.

1. Isolation levels are defined by their bugs

Read the levels bottom-up, as “which anomalies are permitted”:

* postgres RR = snapshot isolation, so phantoms don’t appear on re-read — but SI is not serializable, which is the whole point of Berenson ’95.

Write skew, the one everyone forgets (and your test suite will demonstrate):

 invariant: at least one doctor on call        T1              T2
 oncall = {alice: true, bob: true}       read alice,bob   read alice,bob
                                         (both true)      (both true)
                                         set alice=false  set bob=false
                                         commit ✓         commit ✓
 result: oncall = {} — invariant broken, yet no write-write conflict:
 the write SETS are disjoint; the danger is in the read→write overlap.

2. Three schools of concurrency control

MVCC’s bargain: writes never overwrite — they append a new version. Readers pick the version visible to their snapshot. You pay in space (dead versions) and in a background debt collector (vacuum / GC). Sound familiar? It’s the LSM bargain (topic 4) applied to time instead of keys.

3. Postgres: MVCC on disk

Every heap tuple carries its own visibility metadata (htup_details.h:124–161):

 HeapTupleHeader
 ┌──────────┬──────────┬──────────┬─────────────────────┐
 │ t_xmin   │ t_xmax   │ t_ctid   │ infomask hint bits  │
 │ creator  │ deleter  │ next ver │ XMIN_COMMITTED etc. │
 │ xact id  │ (0=live) │ (chain)  │ (cached clog lookups)│
 └──────────┴──────────┴──────────┴─────────────────────┘
 UPDATE = insert new version + set old tuple's t_xmax + link t_ctid.
 DELETE = set t_xmax. Nothing is removed until VACUUM.

A snapshot is (xmin, xmax, xip[]) (snapshot.h:138–165): all xids < xmin visible, ≥ xmax invisible, in-progress list xip[] invisible. Visibility = pure function of (tuple header, snapshot) — HeapTupleSatisfiesMVCC.

flowchart TD
    A[tuple] --> B{t_xmin committed\nbefore my snapshot?}
    B -- no --> INV[invisible]
    B -- yes --> C{t_xmax set and committed\nbefore my snapshot?}
    C -- yes --> INV2[invisible - deleted]
    C -- no --> VIS[visible]

HOT updates: if no indexed column changed and the new version fits on the same page, skip all index updates — index points at the chain head, readers walk t_ctid within the page. This is why “UPDATE = INSERT+DELETE” is only half true in postgres.

The debt: vacuum. Dead versions accumulate; heap_page_prune_opt does opportunistic per-page cleanup during reads; heap_vacuum_rel does the full pass. XID wraparound is the failure mode that pages DBAs at 3am.

4. In-memory MVCC (Hekaton school)

Hekaton flips the postgres layout: versions live in a chain hanging off a lock-free index; timestamps (begin_ts, end_ts) replace xmin/xmax; commit processing validates reads (serializable OCC over versions); GC is cooperative — every thread cleans as it walks.

Wu/Pavlo VLDB’17 measured the design space (version storage: append-only vs delta vs time-travel; ordering: newest-to-oldest wins; GC: cooperative wins) — read it as a menu with benchmark-backed prices.

5. Code to read (guides in this dir)

Further references: Kung & Robinson “On Optimistic Methods for Concurrency Control” (TODS 1981) — the OCC school’s founding paper (read/validate/write phases; RocksDB’s OptimisticTransaction is this, verbatim); one of the most-cited DB papers of all time.

6. Experiments (experiments/)

src/mvcc.rs — YOU implement MVCC with snapshot isolation over an in-memory KV store. The tests fix the contract:

• snapshots are stable; uncommitted writes invisible; read-your-own-writes
• write-write conflict → first-committer-wins abort
• write_skew_happens_under_si — a test that PASSES when the anomaly occurs (you must be able to produce the bug before you prevent it)
• Mode::Serializable prevents it (track read sets; abort on rw-conflict)
• GC drops versions older than the oldest active snapshot

src/bin/txn_bench.rs — provided; runs once mvcc.rs compiles: threaded throughput of your MVCC vs a single Mutex<HashMap>, read-heavy (95/5) and write-heavy (50/50) mixes. Predict the crossover in notes.md first.

7. M8 checklist (capstone)

• Design the MVCC graph: copy-on-write matrices + versioned reads. Key question: the version unit — whole matrix? tile? delta? (A graph txn touching 1% of a matrix shouldn’t copy 100% of it. Delta matrices from topic 20 ARE pending versions.)
• Single-writer/multi-reader first (FalkorDB’s actual model) — write down what that buys (no write-write conflicts, no validation) and what it costs (write throughput ceiling)
• Then study the reference’s mvcc_graph.rs/cow.rs and diff against your design in notes.md

Isolation levels, made rigorous: history patterns and write skew

Berenson et al.’s SIGMOD ’95 critique is the paper that made isolation rigorous — and, accidentally, the paper that NAMED snapshot isolation and its flaw, seven years before anyone shipped a fix. This chapter replaces ANSI’s ambiguous prose phenomena with precise history patterns, then uses them to place SI in the hierarchy — above repeatable read, incomparable to serializable. Read it before the SSI chapter or that one won’t land.

The setup

ANSI SQL-92 defined isolation levels by three prose “phenomena” (dirty read, non-repeatable read, phantom). The authors show the prose is ambiguous: a strict reading (only forbid the exact anomaly sequence) permits histories everyone agrees are broken; a loose reading over-forbids. Section 3’s move: redefine everything as history patterns over reads/writes/commits/aborts.

The notation to internalize (worth the hour alone)

 P0 dirty write     w1[x] … w2[x]            (both uncommitted)
 P1 dirty read      w1[x] … r2[x]            before c1/a1
 P2 fuzzy read      r1[x] … w2[x]            before c1
 P3 phantom         r1[P] … w2[y in P]       predicate P, not item!
 P4 lost update     r1[x] … w2[x] … w1[x]
 A5A read skew      r1[x] … w2[x] w2[y] c2 … r1[y]
 A5B write skew     r1[x] r1[y] … w2[y] … w1[x]   (your doctors test!)

• The P3 correction is the famous one: ANSI’s phantom was item-based; real phantoms are predicate-based. Locking a row you read doesn’t lock the rows that WOULD have matched.
• The lost-update ladder: ANSI REPEATABLE READ (as literally written) permits P4. Locking-based RR doesn’t. The prose and the implementations had diverged for a decade.

Snapshot isolation, defined and dethroned

Section 4 defines SI: reads from a snapshot, first-committer-wins on writes. Then the twist that structures your whole experiments crate:

• SI forbids P0–P2, P4, A5A — it sits ABOVE ANSI Repeatable Read.
• SI permits A5B write skew — so it’s incomparable to serializable.
• Hence the paper’s hierarchy is a partial order, not a ladder:

            Serializable
             /        \
   SI (no P4, allows A5B)   Repeatable Read (locking; no A5B via locks,
             \        /       allows phantoms P3)
            Read Committed
                 |
            Read Uncommitted

Oracle shipped SI as “serializable” for years. Postgres called it “repeatable read” (honest) and later added SSI on top (next guide).

Questions for notes.md

1. Write the doctors-on-call write skew in the paper’s history notation, and show which forbidden phenomenon it does NOT match (that’s why SI lets it through).
2. Why can’t first-committer-wins catch write skew? (One sentence: the conflict is r→w across txns, not w→w.)
3. Predicate phantoms in a graph: “MATCH (n:Person) WHERE n.age > 40” ran twice in a txn while another txn CREATEs a matching node. Which structure would M8 need to lock/validate — a label matrix? an index range? Is that even expressible as key locks (recall RocksDB guide Q3)?
4. Your mvcc.rs implements exactly Section-4 SI. Which tests map to which phenomena? (Label each test with its P/A number in a comment.)

Done when

You can define SI in one sentence of history notation and name the exact anomaly that separates it from serializable — without looking.

References

Papers

• Berenson, Bernstein, Gray, Melton, O’Neil, O’Neil — “A Critique of ANSI SQL Isolation Levels” (SIGMOD 1995, arXiv:cs/0701157) — ~1.5 h; §3’s history notation and §4’s SI definition are the core

In-memory MVCC: timestamps as locks, and the design-space price list

What does MVCC look like when the disk-era assumptions are deleted? Hekaton (SIGMOD ’13) answers with one design — no locks, no latches, no pages; Wu & Pavlo’s VLDB ’17 evaluation answers with the whole design SPACE, benchmark-backed prices attached. Read Hekaton first (a design), then Wu/Pavlo (the menu).

Hekaton — MVCC with no locks, no latches, no pages

The version record is self-describing, like a postgres tuple but timestamp-based:

 ┌──────────┬─────────┬──────────────┬─────────┐
 │ begin_ts │ end_ts  │ index links  │ payload │
 └──────────┴─────────┴──────────────┴─────────┘
 live version: end_ts = ∞
 during update: end_ts = writer's txn-id (acts as the write lock!)
 visibility: begin_ts ≤ my_read_ts < end_ts

Key moves to internalize:

1. Txn-ids double as locks. Storing a txn-id in end_ts is the write-write conflict check: a second writer sees a txn-id there and aborts/waits. One CAS = lock + version link. (Bit-smuggling again — the id/timestamp distinction is one bit.)
2. Commit processing, not commit point. At commit, get commit_ts, then validate (serializable = re-check read set unchanged + rescan scan predicates), then write log, then fix up all your begin/end_ts fields from txn-id → commit_ts. Readers who hit a txn-id must chase the txn’s state — visibility can depend on an in-flight commit (commit dependencies, taken instead of blocking).
3. Indexes point at version chains; lock-free hash + Bw-tree (topic 9’s protagonists) — MVCC and lock-free structures co-designed.
4. Cooperative GC: any thread that walks past a version older than the oldest active read_ts unlinks it. No vacuum process; the workload cleans itself in proportion to how much it reads.

The visibility test is one range check — except either field may still hold a txn-id, and then the reader chases the writer’s state:

#![allow(unused)]
fn main() {
fn visible(v: &Version, read_ts: u64, txns: &TxnTable) -> bool {
    let begin = match v.begin_ts {
        Stamp(ts) => ts,
        TxnId(id) => match txns.state(id) {
            Committing { commit_ts } => commit_ts, // take a commit DEPENDENCY:
            _ => return false,                     // I abort if the writer does
        },
    };
    let end = match v.end_ts {
        Stamp(ts) => ts,       // superseded at ts
        TxnId(_) => u64::MAX,  // being updated — still the latest for readers
    };
    begin <= read_ts && read_ts < end
}
}

Contrast postgres on every axis: ts vs xid+clog+hint-bits; validation vs SIREAD; cooperative GC vs vacuum; new-to-old chains vs t_ctid old-to-new.

Wu/Pavlo — the menu with prices (VLDB ’17)

They implement every combination in one system and measure. The axes:

The meta-lesson (their words, roughly): everyone argues about CC algorithms, but version storage and GC decide throughput. Storage layer > protocol. (The RUM triangle strikes again.)

Questions for notes.md

1. Hekaton’s end_ts-as-lock: write the CAS-based first-writer-wins in pseudocode. Your mvcc.rs does the same check where? (Point at the line once implemented.)
2. Delta storage wins for writes; append-only N2O for reads. Which is a GraphBLAS delta matrix (topic 20)? So M8’s “copy-on-write + deltas” sits where in the Wu/Pavlo taxonomy — and what does their data predict about its read path?
3. Logical vs physical index pointers: FalkorDB’s node ids ARE logical indirection into matrices. What does that make “index management” cost for a graph MVCC — which updates still have to touch indexes?
4. Cooperative GC in proportion to reads: what happens to a write-only hot key that nobody reads? (Wu/Pavlo call this out — find the fix.)
5. Predict, then check §6 of Wu/Pavlo: at 40 cores, high contention, what ruins MVOCC — validation aborts or timestamp allocation?

Done when

You can fill the 5-axis table from memory and place postgres, Hekaton, and your M8 design in it — one row each.

References

Papers

• Diaconu, Freedman, Ismert, Larson, Mittal, Stonecipher, Verma, Zwilling — “Hekaton: SQL Server’s Memory-Optimized OLTP Engine” (SIGMOD 2013) — ~1.5 h; the version format and commit processing sections carry it
• Wu, Arulraj, Lin, Xian, Pavlo — “An Empirical Evaluation of In-Memory Multi-Version Concurrency Control” (VLDB 2017) — ~1 h; read it as a menu with prices, the tables and §6 graphs carry the message

Postgres MVCC: every tuple carries its own visibility

Postgres stores versions IN the table: each heap tuple’s header names its creator and deleter, and visibility is a pure function of (tuple header, snapshot) — no lock manager consulted on the read path. This chapter walks the tuple header, the snapshot, the visibility function that is the spec of snapshot isolation, the write paths, and the debt collectors that clean up after all of it.

1. The tuple header IS the MVCC state (htup_details.h)

• :124–125 — t_xmin (inserting xact), t_xmax (deleting/locking xact).
• :161 — t_ctid: chain pointer to the newer version of this row. Read the big comment at :86–111: following t_ctid requires re-checking that the next tuple’s xmin equals this tuple’s xmax — the chain can be broken by vacuum, and t_ctid is overloaded for speculative insertion tokens.
• :204–208 — hint bits: HEAP_XMIN_COMMITTED / INVALID, HEAP_XMAX_*. These are a cache of clog lookups written back into the tuple itself by readers. First reader pays the clog probe, everyone after reads a bit. (Reader-writes-metadata: same trick as topic 6’s usage counters.)

2. The snapshot (snapshot.h:138–165, snapmgr.c)

SnapshotData: xmin (:153 — everything below is decided), xmax (:154 — everything at/above is invisible), xip[]/xcnt (:164–165 — in-progress xids at snapshot time, invisible). GetSnapshotData (procarray.c:2114) builds it by scanning the proc array — this scan was postgres’s scalability wall until the 2020 rework (note GetSnapshotDataReuse :2034: if nothing committed since, reuse the old snapshot wholesale).

XidInMVCCSnapshot (snapmgr.c:1869) — the three-way check. Note it’s a binary search over xip for big arrays: snapshot cost scales with concurrent write transactions.

3. The visibility function (heapam_visibility.c)

HeapTupleSatisfiesMVCC :939 — read the whole thing; it is the spec of SI:

1. xmin aborted → invisible; xmin in-progress and not me → invisible
2. xmin mine and cid < my command → visible (read-your-own-writes lives here, via CommandId — statement-level granularity inside a txn)
3. xmin committed but XidInMVCCSnapshot(xmin) → invisible (committed AFTER my snapshot — this is the line that makes it “snapshot”)
4. then the same dance for xmax to decide “deleted yet, for me?”

The same function, minus a decade of hint-bit engineering:

#![allow(unused)]
fn main() {
fn satisfies_mvcc(t: &Tuple, s: &Snapshot) -> bool {
    // "visible xid" = committed AND not still in flight at snapshot time
    let vis = |xid: Xid| committed(xid) && !in_snapshot(xid, s);
    if t.xmin == s.my_xid {
        if t.cmin >= s.cur_cid { return false; } // later command in my own txn
    } else if !vis(t.xmin) {
        return false;                            // creator invisible to me
    }
    match t.xmax {
        None => true,                            // never deleted
        Some(x) if x == s.my_xid => t.cmax >= s.cur_cid,
        Some(x) => !vis(x),                      // deleter invisible ⇒ row lives
    }
}

fn in_snapshot(xid: Xid, s: &Snapshot) -> bool { // committed AFTER my snapshot?
    xid >= s.xmax || (xid >= s.xmin && s.xip.binary_search(&xid).is_ok())
}
}

• HeapTupleSatisfiesUpdate :511 — the OTHER visibility function, used by UPDATE/DELETE to find the latest version and report invisible/being-updated — this is where waiting-on-a-lock and the EvalPlanQual re-check originate.
• SetHintBits machinery :83–112 — even hint-bit writes are batched now (SetHintBitsState): amortize BufferBeginSetHintBits over a page. The amortize-and-batch pattern, again.
• Bonus: HeapTupleSatisfiesMVCCBatch :1690 — visibility checks vectorized over a page. Topic 11 foreshadowing.

4. Write paths (heapam.c)

• heap_insert :2004 — new tuple: xmin = my xid, xmax = 0. Note the WAL record is built AFTER the page change, inside the critical section — topic 5’s reserve-then-copy in action.
• heap_delete :2717 — nothing moves; set xmax, clear some flags. The “delete” is a metadata write.

• heap_update :3201 — the long one. Skim for the shape: HeapDetermineColumnsInfo :3382 (which indexed columns changed?), use_hot_update :3233/:3981 (same-page + no indexed cols changed →

  4029 mark HEAP_HOT_UPDATED, skip index inserts entirely).

 HOT chain (one page):        index entry ──► lp 1 (root, HOT_UPDATED)
                                                │ t_ctid
                                              lp 3 (HEAP_ONLY_TUPLE)
                                                │ t_ctid
                                              lp 5 (HEAP_ONLY_TUPLE) ◄ live
 readers walk the chain under the page latch; prune collapses it later.

5. The debt collectors

• heap_page_prune_opt (pruneheap.c:271) — opportunistic: any reader that notices a prunable page cleans it, no vacuum needed. HOT chains collapse to a redirect line pointer.
• heap_vacuum_rel (vacuumlazy.c:624) / lazy_scan_heap :1279 — the full pass: collect dead TIDs, delete index entries, then mark line pointers reusable. Two-phase because an index entry must never point at a reused slot.

Questions for notes.md

1. Why must the index-entry deletion happen BEFORE line pointers are recycled? Construct the corruption if the order flipped.
2. Hint bits make reads write. Which topic-6 lesson does that complicate (think: checksums, dirty buffers from SELECTs)?
3. A snapshot with 10K concurrent writers makes XidInMVCCSnapshot a binary search over 10K xids per tuple. What does Hekaton’s timestamp design pay instead?
4. FalkorDB angle: postgres stores versions IN the table (old versions inflate the heap). For a graph whose “table” is a sparse matrix, where would old versions live — and is that closer to append-only (postgres) or delta (Hekaton per Wu/Pavlo taxonomy)?

Done when

You can execute HeapTupleSatisfiesMVCC on paper for: (a) my own insert, (b) a commit that landed after my snapshot, (c) a HOT-updated row mid-chain.

References

Code

• postgres — src/backend/access/heap/heapam.c, heapam_visibility.c, src/include/access/htup_details.h, src/include/utils/snapshot.h, src/backend/utils/time/snapmgr.c, pruneheap.c, vacuumlazy.c; ~2.5 h — read HeapTupleSatisfiesMVCC in full first, it is the spec of SI, and the :86–111 comment in htup_details.h before chasing t_ctid

OCC and 2PL, same skeleton: RocksDB transactions

RocksDB ships BOTH optimistic and pessimistic transactions over the same base class — the cleanest side-by-side of the two concurrency schools you’ll find in production code. Both buffer writes privately and differ only in WHEN conflicts are detected: at access time (TryLock) or at commit time (validation).

Everything hangs off sequence numbers: a RocksDB snapshot is just “the seq at begin”. MVCC comes free from the LSM (topic 4): old versions already exist as older entries; a snapshot pins them against compaction GC.

1. Shared skeleton — transaction_base.

• Writes are buffered in a private WriteBatchWithIndex — nothing touches the DB until commit. Reads go through the batch first (read-your-own- writes), then the DB at the snapshot.
• SetSnapshot (transaction_base.h:264) — note snapshot_needed_ :270: snapshots can be taken lazily on first read.
• So: both flavors are “buffer writes, decide at commit”. The ONLY difference is when conflicts are detected.

2. Optimistic — optimistic_transaction.

• CheckTransactionForConflicts (h:67) → TransactionUtil::CheckKeyForConflicts (transaction_util.cc:20) → CheckKey :50.
• The validation trick: for each written key, ask “has this key been written at a seq > my snapshot seq?” — answered from the memtable only (cache_only): if the memtable’s earliest seq is newer than my snapshot, RocksDB can’t know and conservatively aborts (TryAgain). Cheap validation, bought with spurious aborts on long transactions.

#![allow(unused)]
fn main() {
// CheckKey, conceptually: "was this key written after my snapshot?"
fn validate(&self, snap_seq: u64) -> Result<(), Abort> {
    for key in self.write_batch.keys() {
        if self.db.memtable_min_seq() > snap_seq {
            return Err(Abort::TryAgain);  // memtable too young to answer —
        }                                 // abort conservatively, retry
        if self.db.latest_seq(key, /*memtable_only=*/ true) > snap_seq {
            return Err(Abort::Busy);      // someone committed over me
        }
    }
    Ok(())                                // batch → DB, atomically
}
}

• Commit modes (optimistic_transaction.cc:66): CommitWithSerialValidate (h:76) — validate inside the single writer queue (correct by serialization); CommitWithParallelValidate (h:78) — take striped locks on the write set, validate, then write. Same structure as your topic-5 group commit vs per-commit trade.

3. Pessimistic — pessimistic_transaction.{h,cc} + lock/point/

• Every Put/Delete calls TryLock (pessimistic_transaction.cc:1151) BEFORE buffering (:495 — lock first, then base-class write). GetForUpdate takes a read→write lock (:1121).
• PointLockManager (lock/point/point_lock_manager.h:110): striped hash of key → LockInfo (h:26), AcquireWithTimeout :208, deadlock detection via wait-for graph (h:216) with a bounded deadlock-info buffer (h:75–93).
• Commit :681 — locks released only after the write lands: strict 2PL.
• Note what’s locked: keys, not rows — a lock manager over an order-preserving keyspace can’t stop phantoms (no gap/range locks here; contrast innodb). Snapshot validation (SetSnapshotOnNextOperation) is layered on top for repeatable reads.

4. The design plane

 conflict cost paid:   at access time          at commit time
                     ┌────────────────┐      ┌──────────────────┐
 pessimistic 2PL     │ TryLock every  │      │ nothing to check │
                     │ write (+ wait) │      │ (locks held)     │
                     └────────────────┘      └──────────────────┘
 optimistic OCC      │ nothing        │      │ CheckKey per     │
                     │ (buffer only)  │      │ written key      │
                     └────────────────┘      └──────────────────┘
 contention ↑ ⇒ OCC abort rate ↑ (wasted work); 2PL queue depth ↑ (waits).

Questions for notes.md

1. Why can OCC validation use the memtable only? What property of LSM seq numbers makes “not in memtable ⇒ too old to conflict… unless memtable is too young” sound — and what does the TryAgain path cost a retry loop?
2. The pessimistic lock manager stripes by key hash. What’s the pathology for a graph workload where every txn touches the same super-node’s adjacency entries?
3. Neither flavor validates READ sets by default — so what isolation do you actually get, and where does write skew sneak in?
4. FalkorDB angle: GRAPH.QUERY writes are single-threaded today (one writer). If M8 keeps single-writer, which of these two machineries do you still need? (Hint: none for w-w; what about r-w validation for serializable reads?)

Done when

You can explain, with file:line, where each school pays its conflict cost, and why both can share one write-buffering base class.

References

Code

• rocksdb — utilities/transactions/: transaction_base.{h,cc} (shared skeleton), optimistic_transaction.{h,cc} + transaction_util.cc (OCC), pessimistic_transaction.{h,cc} + lock/point/point_lock_manager.h (2PL); ~1.5 h

SSI: serializable snapshot isolation without blocking anyone

How postgres turned SI into SERIALIZABLE with passive markers instead of blocking locks — Ports & Grittner’s VLDB ’12 account of productionizing Cahill’s dangerous-structure theorem. Prereq: the Berenson critique (reading-ansi-critique.md) — you need write skew cold.

The theory in one diagram

Every SI anomaly contains a dangerous structure: two consecutive rw-antidependencies with the pivot in the middle:

        rw            rw
  T_in ────► T_pivot ────► T_out        rw edge: T reads x, then U writes x
                                        (U "un-reads" T's snapshot)
  … and T_out commits FIRST of the three.

Cahill’s theorem: every non-serializable SI execution has this shape. So: track rw-antidependency edges; when a txn accumulates BOTH an inbound and an outbound rw edge (it became a pivot), abort somebody. This is conservative — some aborted histories were actually fine (false positives) — but it never misses a real cycle.

The whole detector, conceptually — two flags per transaction and one rule:

#![allow(unused)]
fn main() {
fn on_rw_antidependency(reader: TxnId, writer: TxnId, g: &mut ConflictGraph) {
    g[reader].out_rw = true;             // reader ──rw──► writer
    g[writer].in_rw = true;
    for t in [reader, writer] {
        if g[t].in_rw && g[t].out_rw {   // t became a pivot:
            abort_someone(t, g);         //   T_in ─rw─► t ─rw─► T_out
        }                                // conservative: false positives yes,
    }                                    // missed cycles never
}
}

Doctors write skew as the structure: T1 reads bob’s row (later written by T2) ⇒ T1 ──rw──► T2; T2 reads alice’s row (later written by T1) ⇒ T2 ──rw──► T1. A cycle of length 2 — each txn is a pivot.

What the paper adds over Cahill (§4–§7)

1. SIREAD locks — not locks at all: passive markers “I read this”, at tuple/page/relation granularity with escalation under memory pressure (coarser = more false aborts, never wrong results). Predicate reads are handled by locking the read RANGE via index pages — this is the answer to phantoms that key-based OCC (RocksDB Q3) can’t give.
2. Commit ordering refinement — only abort when T_out committed first (fewer false positives than raw Cahill).
3. Safe snapshots & DEFERRABLE — a read-only txn can prove it can never be part of a dangerous structure and drop ALL tracking; RO backups run at serializable for free (after possibly waiting).
4. Memory bounding — SIREAD state must survive commit (rw edges can form after you commit!) and is only cleaned when overlapping txns end; §7’s summarization is the price of bounded RAM.
5. 2PC interaction, and why prepared transactions make everything worse.

The costs (§8, the honest part)

• ~7% overhead on their benchmarks at low contention; abort rate is the real cost and it’s workload-shaped (hot rw pairs → pivot storms).
• Retry is the application’s job: serialization_failure (40001) means “run it again”, so SSI only works if the app loops.

Questions for notes.md

1. Why must SIREAD locks outlive commit? Construct the history where the dangerous structure completes after the reader committed.
2. Lock escalation trades memory for false aborts. Where’s the same trade in your mvcc.rs Serializable mode (hint: your read-set granularity is whole keys — what’s the graph equivalent of escalating to a relation)?
3. Read-only txns: why can they NEVER be T_pivot? (Which edge can’t they have?) How does that justify the safe-snapshot optimization?
4. M8: FalkorDB is single-writer. With exactly one writer at a time, can a dangerous structure form at all between two write txns? Between a writer and concurrent readers? So is SSI machinery needed, or does single-writer + SI already equal serializable? (Prove it with the pivot definition — this is the M8 design shortcut.)

Done when

You can draw the dangerous structure from memory, place both write-skew txns on it, and answer Q4 — it decides how much of this paper M8 needs.

References

Papers

• Ports & Grittner — “Serializable Snapshot Isolation in PostgreSQL” (VLDB 2012, arXiv:1208.4179) — ~1.5 h; §4–§7 are the production engineering, §8 the honest costs
• Cahill, Röhm, Fekete — “Serializable Isolation for Snapshot Databases” (SIGMOD 2008) — the dangerous-structure theorem this paper productionizes; the theorem statement is enough

The minimal transactional KV interface: surrealdb’s kvs layer

surrealdb doesn’t implement MVCC — it abstracts over engines that do (tikv, foundationdb, rocksdb, in-memory…), which forces it to define the minimal transactional interface a multi-model DB needs. Read this one for ARCHITECTURE, not algorithms: that interface is a good checklist for M8’s storage-backend abstraction (M1).

1. The layering

 Datastore (ds.rs) ── transaction() :3353 ──► Transaction (tx.rs:94)
                                                 │ caching + typed keys
                                                 ▼
                                              Transactor (tr.rs:37)
                                                 │ uniform async KV-txn API
                                                 ▼
                                   engine flavor (mem/rocksdb/tikv/fdb…)

• TransactionType (tr.rs:15): just Read | Write — declared UP FRONT at begin. Compare postgres (any txn can write) — declaring intent enables single-writer engines and read-only fast paths.
• LockType on Datastore::transaction() (ds.rs:3353): Optimistic | Pessimistic — the CHOICE of school is a per-transaction parameter passed down to engines that support both. The two RocksDB flavors you just read are literally behind this flag.
• TransactionFactory (ds.rs:314) / builder plumbing (ds.rs:450–571): the multi-backend dispatch. M1’s StorageBackend trait, grown up.

2. The Transactor API (tr.rs) — read the signatures

• get/getm/getr/getp (:119–155) all take version: Option<u64> — versioned point-in-time reads are part of the public KV contract, not an engine internal. (Only some engines honor it; capability, not guarantee.)
• set :166 vs put :190 vs putc :202 — put fails if the key exists; putc is compare-and-set on the current value: optimistic concurrency primitives exposed as API, so upper layers can do OCC over any engine.
• commit :103 / cancel :95 — commit is where engine-level conflict errors surface; the query layer retries.

3. Transaction (tx.rs:94, impl :693)

Wraps Transactor with typed keys and read-through caches. The thing to notice: caching inside a transaction is trivially correct — the snapshot is immutable, so a within-txn cache never invalidates. (Topic 6’s hardest problem — invalidation — deleted by MVCC.)

Questions for notes.md

1. version: Option<u64> on every read: what does time-travel-as-API cost the engines that support it (GC can’t drop what an API can name)?
2. Read/Write declared at begin: what optimizations does that unlock for a single-writer engine? What does FalkorDB’s GRAPH.RO_QUERY vs GRAPH.QUERY split already encode?
3. putc (CAS) as the portable OCC primitive: sketch how you’d build first-committer-wins snapshot isolation on top of ONLY get/putc.
4. M1 retrospective: does your storage-backend trait from topic 1 admit a transactional backend, or did you bake in auto-commit? What would you change now?

Done when

You can list the 6–8 operations a transactional KV interface needs to support a multi-model DB, and say which are capabilities vs guarantees.

References

Code

• surrealdb — surrealdb/core/src/kvs/: ds.rs, tr.rs, tx.rs; ~1 h — read the Transactor signatures in tr.rs, they ARE the interface checklist

Topic 8 notes — transactions & MVCC

Predict FIRST, then measure.

Predictions (fill in BEFORE running txn_bench)

Reasoning space:

• The mutex serializes even pure readers; MVCC readers only touch the metadata lock briefly per op. But your MVCC pays per-version allocation
  • retry loops. At what abort rate does retrying erase the win?
• 4 threads, 4 ops/txn, 64 keys, 50% writes → estimate P(two concurrent txns share a written key) before you look at the abort column.

Implementation log (mvcc.rs design decisions)

• Version storage: append-only newest-to-oldest? Where does your design sit in the Wu/Pavlo 5-axis table? (Fill the row.)
• What exactly does your metadata mutex protect, and what would it take to shard it (topic 7’s SHARDS trick — does it apply cleanly here)?
• Label each test with its Berenson phenomenon (P1/P4/A5A/A5B) in a comment — the ansi-critique guide asks for this.
• Serializable = backward read-set validation. Construct one history your validator aborts that SSI would have allowed (a false positive).

Questions — reading-postgres-heapam.md

1. Index-entry deletion before line-pointer recycling: the corruption if flipped?
2. Hint bits make reads write — which topic-6 consequence?
3. XidInMVCCSnapshot at 10K writers vs Hekaton timestamps: costs?
4. Where do old versions live in a matrix-backed graph — append-only or delta?

Questions — reading-rocksdb-transactions.md

1. Why is memtable-only OCC validation sound (and when does it TryAgain)?
2. Striped key locks vs a super-node’s adjacency: the pathology?
3. No read-set validation by default — what isolation, where does skew enter?
4. Single-writer M8: which machinery survives?

Questions — reading-ansi-critique.md

1. Doctors write skew in history notation; which phenomenon does it evade?
2. Why first-committer-wins can’t catch write skew (one sentence).
3. Predicate phantoms for MATCH — label matrix? index range? key locks?
4. Test-to-phenomenon mapping done in mvcc.rs comments?

Questions — reading-ssi-postgres.md

1. History where the dangerous structure completes after the reader commits?
2. Your read-set granularity vs SIREAD escalation — the graph equivalent?
3. Why can a read-only txn never be the pivot?
4. The M8 shortcut: single-writer + SI — prove (with the pivot definition) whether it’s already serializable. Write the argument here:

Questions — reading-inmemory-mvcc.md

1. end_ts-as-lock CAS pseudocode; the equivalent check in your mvcc.rs is at line ___.
2. Delta matrices in the Wu/Pavlo taxonomy; predicted read-path cost?
3. Logical node-ids as indirection: which graph updates still touch indexes?
4. Write-only hot key vs cooperative GC — the fix?
5. MVOCC at 40 cores: validation aborts or ts allocation? (Predict, then check §6.)

The 5-axis placement (fill after all readings)

M8 log (capstone milestone)

• mvcc.rs passes all 8 tests (write skew demonstrated AND prevented)
• txn_bench run; all predictions scored above
• MVCC graph design written: version unit (matrix/tile/delta), CoW granularity, reader visibility rule
• single-writer/multi-reader argument from SSI Q4 recorded — decides whether M8 needs validation at all
• reference’s mvcc_graph.rs / cow.rs studied; diff vs my design noted

Topic 9 — Concurrency: Latches, Lock-Free & Epochs

Scaling across cores is where the hardest bugs and the biggest wins live. Topic 8 asked “who sees what” for transactions; this topic asks the same question for NANOSECONDS: two threads, one cache line, who wins?

Budget: ~12 h. Order: §1 vocabulary → §2 memory ordering → §3 latch protocols → §4 reclamation → §5 code → experiments → M9.

1. Latches vs locks (say it right)

Everything in this topic is latches. The three escalation rungs:

 mutex/rwlock          →  optimistic (version check)  →  lock-free (CAS)
 block on conflict        restart on conflict            never block; help
 postgres LWLock          LeanStore HybridLatch (T6)     RocksDB memtable,
                          + OLC B-trees                  crossbeam SkipSet

2. Memory ordering in one table (Rust atomics)