turso’s simulator: every failure is a u64 seed

The most readable production DST codebase in Rust: seeded clock, fault-injecting IO, metamorphic properties, and a shrinker, all in one testing/simulator/ tree. Read it as the reference implementation for our dst.rs stub and for M16 — every piece here has a miniature counterpart in the experiments.

Layout

 testing/simulator/
   main.rs          entry: seed → config → plan → execute → check
   runner/
     clock.rs       SimulatorClock — time is an RNG stream
     io.rs          SimulatorIO — fault injection switchboard
     file.rs        SimulatorFile — per-op faults + seeded latency
     execution.rs   drive the plan, catch assertion failures
     doublecheck.rs run the same plan twice, diff outputs
     bugbase.rs     known-bug corpus (regression seeds)
   generation/      plan/property/query generators
   model/           the in-memory oracle + interaction model
   shrink/          plan minimization

Anchor map

1. Time is an RNG stream (clock.rs:25)

Every now() call advances the clock by random_range(min_tick.. max_tick) — no wall clock anywhere.

#![allow(unused)]
fn main() {
// time is data: every now() consumes seeded randomness and ADVANCES
struct SimClock {
    curr: Duration,
    rng: ChaCha8Rng,                 // portable, versioned — never the default RNG
    min_tick: Duration,
    max_tick: Duration,
}

impl SimClock {
    fn now(&mut self) -> Instant {
        self.curr += self.rng.random_range(self.min_tick..self.max_tick);
        Instant::from(self.curr)     // monotone progress: timeout loops terminate
    }
}
}

Question: why must now() ADVANCE time rather than return a fixed value? (What loops forever if time never moves? Think timeout code.)

2. Fault injection lives in the FILE (file.rs)

Not “kill the process” — per-operation faults: a pwrite can fail, a sync can fail, any op can be delayed and reordered via the DelayedIo queue. This is the fault model our sim_fs.rs copies (buffered-until-sync + tear-on-crash). Question: which topic 5 crash-matrix cell does each of {pwrite fault, sync fault, delayed write + crash} correspond to?

3. Properties = metamorphic oracles (generation/property.rs)

SelectSelectOptimizer is TLP-shaped: two spellings of the same query must agree. ReadYourUpdatesBack is a session guarantee (DDIA ch. 5 — same anomaly, single node). DoubleCreateFailure pins error-path behavior. Note the generation trick: unrelated random queries are interleaved WITHOUT breaking property invariants — coverage and oracles coexist.

4. Doublecheck (runner/doublecheck.rs)

Run the identical plan twice; outputs must match byte-for-byte. This is the cheapest oracle of all: it needs NO model — it only needs determinism. Question: what class of bug does doublecheck catch that the model oracle misses? (Hint: iteration order, uninitialized memory, hidden wall-clock reads.)

5. The bug base (runner/bugbase.rs)

Found bugs persist as seeds — the regression suite is a list of u64s. Compare: our topic 15 sim tests hardcode seeds 42/7/11/13.

Questions for notes.md

1. ChaCha8 everywhere, not the default RNG — why does DST need a portable, versioned RNG? What breaks on rand upgrades?
2. inject_fault_selective targets file stems (WAL vs db file) — which bug class needs faults on ONE file only?
3. Where does turso’s simulator sit vs Antithesis (whole-VM determinism)? What can each test that the other can’t?
4. The shrink/ module: why is shrinking HARDER for stateful op sequences than for pure inputs (proptest’s integrated shrinking vs delta debugging)?
5. For M16: which three properties from generation/property.rs port directly to Cypher? Sketch the graph equivalents.

References

Code

• turso — testing/simulator/ (clock/io/file fault injection, interaction plans, properties, doublecheck, shrink) plus fuzz/fuzz_targets/expression.rs for structured fuzzing via arbitrary — clone it; the anchor map above is your reading order