deterministic simulation testing #
a deterministic simulation runs the production code of a whole service in one process, with everything outside that process replaced by something a seeded generator controls: the clock, the randomness, when background work runs, which storage call fails, when a message arrives, and when the process dies. a failure is then a history that replays exactly and shrinks, including failures that depend on timing. it is stateful property testing with the generator promoted from choosing operations to choosing the order and the weather in which they happen.
it earns its cost on code whose bugs need a coincidence: two requests interleaved at one statement, a lease expiring between a check and a write, a message delivered twice, a restart between a commit and its acknowledgement. none of those show up when one client makes one request at a time against a healthy machine.
what the simulator owns, and what stays real #
keep the inside real: the request router, the handlers, the background services, the task queue, and an actual database engine. mocking the database buys speed and gives up its semantics, which are usually what the bug depends on.
take ownership of every way the process touches the outside:
- time and randomness. code that receives its clock, sleep, and random bytes through an interface can be handed a simulated one with no change to its logic; code that reads the system clock directly has to be refactored first. a single process can hold more than one such interface (one for request handlers, another for locks and the connection pool), and all of them have to be swapped together or part of the system keeps real time.
- background work. a service that wakes on a timer becomes a rule: "run the scheduler once, now". nothing runs on its own. this is what makes a history a complete description of the run.
- the storage chokepoint. if every statement passes through one function, that function is two seams at once: a place to fail a statement, and a place to stop this request and let another one run.
- message delivery. a broker in simulation mode starts no consumer thread. the generator decides when each queued message is handed over, and can hand it over late, twice, out of order, or never.
- the network edge. build requests in memory and call the router. no sockets.
- death. a crash drops everything held in memory and keeps what the database committed.
the checker's own reads must be exempt from faults. the invariants read the database through the same chokepoint the faults are injected at. without a switch that suspends injection while the checker looks, the harness fails its own assertions and reports them as findings.
races as generated choices #
a race found by running threads freely is a failure that happens sometimes. in the simulator, a rule runs two requests on two threads that share a baton: only the holder moves, and the baton changes hands only at the statements a generated schedule names. same schedule, same interleaving. a lost update then has a seed, replays every time, and shrinks to the two requests and the one switch that matter:
run is SCHEDULED
A: read run B: read run
A: decide -> RUNNING B: decide -> PENDING
A: write B: write
both accepted; history reads RUNNING then PENDING
the repair that held up: guard the write on the state the decision was made against (update ... where state_id = :read_id), and on a miss re-read and decide again, with a bounded number of tries. the loser then decides against the winner's state instead of overwriting it. after that change the real binary under free threads committed 0 bad histories in 300 trials, against 73 for the original with its row lock.
the schedule has to respect what the database allows. with a single-writer engine, a request that yields inside its own write transaction leaves the other waiting on the lock forever, so switches are offered at statements outside a transaction and at its start, never within.
confirm a simulated race against the real binary with free-running threads before and after the fix. the simulator's interleaving is at statement granularity, and the claim that matters is about the shipped program.
switch points have to be drawn often enough to matter. a planted bug in a release path survived until the generator was biased toward switching at the statements around a decrement. an interleaving rule that rarely switches is the same failure as a generator that never builds an interesting state.
oracles for a whole service #
invariants that need no second opinion carry the most: a counter never goes below zero, slots in use equal the leases holding them, a refused request changes nothing, a record's history is a legal path through its state rules, an accepted message is persisted exactly once however many times it is delivered.
the original implementation, when the subject is a rewrite or a port. record each generated history as a sequence of requests with the state after each one, replay it against the original server, and compare. every disagreement is one of four things, and all four turn up:
- a bug in the rewrite. most of them.
- a bug in the original. the rewrite inherited the design, so the simulator finds it in both; these reproduced against the original with real concurrency at rates between a third and nine in ten.
- a bug in the harness. timestamps from before simulated time began were compared untranslated, and the original ran a periodic sweep the replay had not accounted for.
- a difference someone chose.
a third, independent model of the rule being checked makes the verdicts faster: when two of three agree, the odd one out is where to look.
porting drops the guarantees that have no visible behaviour. the original serialized state changes with a row lock, one line under a comment about races. the port lost it, and no ordinary test noticed, because a lock changes nothing on the path where one client acts alone. when porting, list what the original guarantees under concurrency and failure, and test for those specifically.
replay has edges. the original is not deterministic: it settles asynchronously after a commit, so the replay needs a pause before it reads. behaviour that depends on elapsed time cannot be replayed at simulated speed against a server on a real clock; cover it with a few targeted probes that run in real time.
bugs that are never wrong at any one step #
an expired lease that never returns its slot violates no invariant at any step. each state is legal; the system just never recovers. these need a rule that removes the pressure and waits: advance the clock past every deadline, run the cleanup services, then assert that nothing is still held. without that rule a simulator checks safety and is silent about leaks, stuck work, and alerts that fire once and never again.
fault scope #
inject the faults the design claims to survive, and treat the rest as noise. a failed statement, a duplicated message, and a restart are in scope for a server that claims to be correct across them. losing committed rows is not, and neither is losing queued messages from a broker documented as in-memory: the simulator reports those separately instead of counting them.
failed statements find a specific and common bug: a read that fails is handled as "not found". a limit that could not be read was treated as a limit that did not exist, and the request was admitted without one.
determinism is itself under test #
the same seed must give the same trace, and that is worth asserting across thousands of cases, not assuming. a failure that does not recur from its seed means something outside the generator's control is still running: a real thread, a wall-clock read, an unseeded id. in a simulator that is a defect to fix before anything else, because every later finding depends on it.
what it costs and what it cannot see #
speed. a simulated service is slow next to a simulated data structure: a few hundred steps a second with real sql underneath, against hundreds of thousands for an in-memory store. logic bugs do not need volume (planted ones were usually caught within 2 to 20 cases) but anything that depends on a rare coincidence the generator does not steer toward will not appear.
reach. the generator finds only what it can build. a trigger kind the generator never created had a bug no run could see until a hand-written probe hit it and the generator learned to build that kind; three more bugs in the same subsystem came from reading the original's evaluator beside the port. when a subsystem lands in the simulator, read the original's code paths against the rule table and list what nothing generates.
scale. a simulated world is small. a queue that never fills at simulated sizes hides the deadlock that happens when it does: one such bug passed every simulated suite and reproduced in 90 seconds against the real network, because simulated payloads were about 4 KiB and real ones about 10 MB. write the simulator's blind spots down and keep one tier that runs the real binary against real dependencies at realistic size.
fidelity. a fake that never fails the way the real dependency fails tests the wrong thing. the pieces left real (the database engine, the task queue) are the ones whose behaviour the simulator can be trusted about.
upkeep. the simulator is a second program that follows the first. a seam that production code stops going through is a hole nobody sees, so the seams belong in the production code's own structure, not in test-only patches.
related #
- stateful property testing — generators, calibration with planted bugs, and choosing an oracle; all of it applies here
- leases and expiry — the slot and lease bookkeeping bugs these races and restarts found
- background tasks — at-least-once delivery, the contract that makes duplicate delivery a fault worth generating
- embedded redis — an in-process engine is what lets a task queue run inside the simulation
sources #
- prefect-server —
src/dst/world.zig(what the simulator owns),src/dst/machine.zig(rules and checks),dst/replay.py(replay against Python Prefect 3.8.2),dst/mutants.sh,dst/findings/(028–030 for the triggers the generator did not build), commitfae0d35(the guarded write), 2026-10-12 - stream —
docs/invariants.md, the deadlock that simulated repository sizes could not reach, 2026-10-09 - Simulation Testing for Liveness, TigerBeetle — separating safety checks from "does it recover once the faults stop"
- Simple Testing Can Prevent Most Critical Failures, Yuan et al., OSDI 2014 — error handling of explicitly signalled failures as the dominant source of catastrophic bugs