Series

Deterministic simulation testing in Go

Bugs in distributed systems often fail a test once and never again. This series is about catching them in Go so that every failure replays exactly from a seed. It follows faultline, the open-source deterministic simulation testing tool I'm building, from the problem to real runs against etcd/raft.

Oct 13, 2026
Oct 13, 20269 min read19 views
A Go test that passed 40,000 times, then lost a write under load
Why timing bugs in distributed systems hide from normal tests, and how deterministic simulation testing makes them replayable
9 min read19 views
Oct 19, 2026
Oct 19, 202610 min read10 views
Cutting etcd/raft failover from 1.25 s to 343 ms, checked against 2,000 simulated leader crashes
A shorter election timeout tested under faults in faultline v0.1.1, with the one seed that took 1.78 seconds replayed
10 min read10 views

Posts in this series

A Go test that passed 40,000 times, then lost a write under load

Cutting etcd/raft failover from 1.25 s to 343 ms, checked against 2,000 simulated leader crashes