Topic 22 — Standard Benchmarks: TPC-H, TPC-C, YCSB, LDBC & Friends

Benchmarks are engineering tools only if you know what each query actually stresses. This topic maps the standard suites to their choke points, builds the two most-copied pieces of benchmark machinery (the YCSB Zipfian generator, TPC-H Q1/Q6), and sets up M22’s standing regression suite.

The map

            OLTP ◄──────────────────────────► OLAP
  TPC-C  YCSB   TATP  SmallBank │ SSB  TPC-H  TPC-DS  ClickBench
  (txn    (KV                   │ (star (choke (100+q,  (one wide
  contention) mixes×skew)       │ schema) points) skew)   table)
                                │
  graph: LDBC SNB interactive (OLTP-ish) / BI (OLAP) / Graphalytics
  vector: ann-benchmarks (recall vs QPS — topic 14)
  real cardinalities: JOB (topic 10 — built because TPC-H is uniform)

graph TD
    Q["a benchmark number"] --> W["workload: mix + distribution<br/>(YCSB: A-F × zipfian/uniform)"]
    Q --> D["data: scale factor + skew +<br/>correlation (dbgen: none!)"]
    Q --> H["harness: open vs closed loop,<br/>think times, warmup, driver cost"]
    Q --> M["metric: tpmC? geomean?<br/>p999? GB/s? recall@10?"]
    W & D & H & M --> V{"change ANY one<br/>⇒ different number"}

Choke points, one line each

• TPC-H Q1: tiny group domain ⇒ hash table free ⇒ pure expression eval + fused agg (our q1_flat makes it explicit).
• TPC-H Q6: 2%-selective scan ⇒ SIMD predicates, the “GB/s” headline query (our q6_branchless, topic 17’s filter shapes).
• TPC-H Q9: 6-way join order + LIKE ‘%green%’ + skew — the optimizer punisher (topic 10).
• TPC-C: the D_NEXT_O_ID hot counter + 1% remote warehouses + think times nobody runs — contention by design, not throughput.
• YCSB: mix × distribution factoring; θ=0.99 zipfian is the whole personality (see the generator math in the reading guide).
• LDBC SNB: correlated power-law datagen + choke points for graphs (topic 13’s guide) — M22’s centerpiece.

Measured baselines (bench_suite, M3 Pro, single thread)

Q6’s oracle at 15.7 GB/s is already half of memory bandwidth with branches — predict what branchless + autovec adds at this 2% selectivity before implementing (topic 17 says: maybe nothing!).

Benchmarking sins checklist (see topic 0’s reading-fair-benchmarking.md)

1. closed-loop tails quoted as latency (coordinated omission)
2. warm cache vs cold unstated; SF that fits in LLC
3. “TPC-C” without think times ⇒ you measured a latch
4. geomean over arithmetic mean (or vice versa) chosen post hoc
5. comparing your tuned build vs their defaults (Fair Benchmarking’s core sin)
6. uniform data standing in for skewed reality (dbgen’s lie; JOB’s reason to exist)

Reading guides

• reading-boncz-tpch.md — TPC-H decoded: 22 queries, 28 choke points
• reading-ycsb.md — YCSB: six mixes, five distributions, one Zipfian generator
• reading-oltpbench-tpcc.md — TPC-C: contention by design (and the harness that runs it honestly)
• reading-duckdb-tpch.md — dbgen as a table function: shipping a benchmark inside the engine
• topic 0: reading-fair-benchmarking.md (DBTest ’18) — methodology
• topic 13: reading-ldbc-snb.md — SNB datagen + interactive driver
• topic 10: reading-leis-vldb15.md analogue — JOB, real cardinalities

Experiments

M22 checklist

• standing suite: LDBC SNB interactive + graph micro-benches + ann-benchmarks recall/QPS, one command, results as data files
• regression tracking across milestones (M0 baselines are the floor; every M* run appends, plots trend)
• three-way shootout: falkordb-scratch vs falkordb-rs-next-gen vs FalkorDB — same driver, same data, same machine, fair- benchmarking checklist applied