db_bench: the shared vocabulary of storage benchmarking

fillseq, readrandom, readwhilewriting — these workload names started in LevelDB, were extended by RocksDB, and now appear in every LSM paper since. This chapter is a skim route through the 10,000-line tool that defines them: the goal is the vocabulary and the measurement shape, not the harness code. Name your own benchmarks in this language and your numbers become comparable to two decades of published results.

Skim route (30–60 min)

The vocabulary worth memorizing

Under every workload name sits the same skeleton — pick an integer, format it as a fixed-width key (GenerateKeyFromInt :3802, WriteMode :5869, KeyGenerator :6088):

#![allow(unused)]
fn main() {
// every fill* workload reduces to how the next integer is chosen
fn next_key(mode: WriteMode, i: u64, n: u64, rng: &mut Rng, perm: &[u64]) -> Key {
    let int = match mode {
        WriteMode::Sequential   => i,                // fillseq: in-order, no
                                                     //   compaction debt
        WriteMode::Random       => rng.next() % n,   // fillrandom/overwrite:
                                                     //   duplicates → garbage
                                                     //   → compaction pressure
        WriteMode::UniqueRandom => perm[i as usize], // pre-shuffled permutation:
                                                     //   random order, no dups
    };
    generate_key_from_int(int)                       // zero-padded fixed width
}
}

• fillseq — sequential-order load. Fast path for an LSM (no compaction debt); papers use it to build the DB before the real test.
• fillrandom / overwrite — random inserts vs. random overwrites of existing keys (overwrite creates garbage → compaction pressure — different beast).
• fillsync — one fsync per write, N/1000 ops: measures durability cost, not throughput.
• readrandom / readseq / readreverse — point lookups vs. iterator scans.
• readwhilewriting — 1 writer + N readers: the “does compaction wreck my read tail?” test. The *whilemerging/*whilescanning variants isolate other interference.
• seekrandom — iterator Seek cost (touches every level; very different profile from Get).
• multireadrandom — MultiGet batching.
• mixgraph (4133) — the odd one out: models Facebook’s measured production distributions (the “Characterizing, Modeling…” FAST’20 paper) with two-term-exponential key ranges (keyrange_dist_a..d) and Pareto value sizes. The industrial answer to “uniform random keys are the wrong distribution” — same motivation as the capstone’s Zipfian workload crate.
• Standard invocation shape: db_bench --benchmarks=fillseq,readrandom --num=10000000 --value_size=100 --histogram — comma list runs in order against the same DB, so earlier benchmarks create the state later ones measure. That ordering is the methodology.

What to notice about measurement

flowchart TD
    F["--benchmarks=fillseq,readrandom,--histogram<br/>comma list runs IN ORDER against the same DB"]
    F --> RUN["Benchmark::Run() (4030)<br/>workload name → method pointer"]
    RUN --> RB["RunBenchmark(n, name, method) (4583)<br/>spawn N threads"]
    RB --> T1["thread 1<br/>Stats + HistogramImpl (2436)"]
    RB --> T2["thread 2 ..."]
    RB --> TN["thread N"]
    T1 --> M["merge per-thread histograms (2488)<br/>never average percentiles — Tene's rule"]
    T2 --> M
    TN --> M
    M --> OUT["default output: throughput<br/>latency histogram only with --histogram"]

• Per-op latency goes through FinishedOps (2564) into a plain HistogramImpl — reported only with --histogram. Default output is throughput (ops/s, MB/s).
• It’s a closed loop like redis-benchmark: each thread issues the next op after the previous completes. There’s a --benchmark_write_rate_limit/read-rate variant for paced writes, but no coordinated-omission correction — same critique applies.
• db_bench measures the embedded engine (no network), so “latency” here is service time by construction — legitimate for engine work, misleading if quoted as user latency.

Takeaway

db_bench’s value is the workload taxonomy, not the harness. When topic 4 (LSM) and M4 (backend shootout) arrive, name capstone benches in this vocabulary (fillseq, readrandom, readwhilewriting) so numbers are comparable against published RocksDB results.

References

Papers

• Cao, Dong, Vemuri, Du — “Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook” (FAST 2020) — the measured production distributions behind mixgraph; optional, skim §4-5

Code

• rocksdb tools/db_bench_tool.cc (~10,400 lines, shallow clone @ 7c80a5a) — do not read this linearly; it’s a flag-driven monolith — follow the skim route table above (30–60 min)