Escapement · deterministic simulation · results

What the fuzzer found.

Four real defects in this Raft implementation, one rule of the paper I had skipped, and three false positives in my own checkers — each one reproducible from a single number. Every bug below is still switchable, so the links reproduce the failure live in the visualizer rather than describing it.

The sweep

100,000 seeds, 40,000 virtual ticks each, eight threads, about eighty seconds. Every seed picks its own cluster size, network, fault schedule, disk behaviour, clock skew, batch size, workload and retry policy — all derived from the seed, so each line below is reproducible from a single number.

violations	0
histories linearizable	100,000 of 100,000 (0 undecided)
elections	3,427,544
log truncations	113,788 (157,991 entries discarded)
crashes / restarts	810,375 / 794,953
torn steps	78,708
snapshots taken / shipped	760,114 / 156,495
membership changes	31,710
reads via ReadIndex	2,247,665 (never entering the log)
messages sent	504,945,002
dropped / partitioned	30,946,296 / 26,485,276
duplicated / reordered	10,662,574 / 1,803,435
client operations	10,698,594 (256,790 never answered)
highest term reached	207

A clean sweep is worth nothing without evidence that it reached anything. The 113,788 truncations are the number that matters: logs actually diverged and were actually forced back into agreement, 113,788 times. The 256,790 unanswered operations matter for the same reason — that is the path where a linearizability checker most easily becomes unsound, and it was exercised on real data rather than only in handcrafted tests.

Bugs it found in this implementation

These are not hypotheticals or planted examples. Each was a real defect in this Raft implementation, found by the simulator during development. Each is still switchable, so the links below reproduce the failure live. Press play at 1000x; each one fails within about six thousand ticks, a few seconds of watching.

1. Index 0 stops being an anchor once you compact §7

term_at(0) returned Some(0) as the "before the beginning of the log" sentinel. That is what lets a leader replicating from scratch pass the prevLogIndex check with no special case, and it is correct — right up until the log is compacted.

A leader whose nextIndex for some follower had walked back to 1 would probe with prevLogIndex = 0. A follower that had compacted through index 8 answered Some(0), the consistency check passed, and the follower appended entries 1..7 onto a log that began at index 9. The result was a silently corrupt log: slot 10 held an entry claiming to be index 1.

Found on seed 11278 as a Log Matching violation. Minimized to zero faults — compaction alone reproduces it, no partition or crash required.

The fix makes the sentinel uniform: the anchor is always last_included, which is (0, 0) for a log that has never been compacted. Index 0 answers on a fresh log and stops the moment anything is discarded.

Reproduce it live → press play; fails by tick 5,780

2. The snapshot and the log are two artifacts that can disagree §7

Installing a snapshot persists the snapshot and then resets the log. I had written a comment claiming the order was safe because a crash in between "leaves the snapshot stored and the log merely un-trimmed — redundant, not lossy."

That is true for the compaction case, where the log is a superset of the snapshot. It is false for the reset case, where the surviving log is a stale, shorter one. Recovery then paired a snapshot at index 120 with a log ending at 119, and the node came back with commitIndex past the end of its own log.

Found on seed 2367, and only reachable because torn steps are modelled: a crash between two durable writes is the entire failure.

Recovery reconciles the two. Fixing it only in memory made things worse in an instructive way — the node's log got repaired while the durable one did not, so the very next append landed at an index the disk was not expecting. The reconciliation is a recovery action and has to be written back to disk.

Reproduce it live → press play; breaks Commit Index Within Log by tick 5,916

3. Two read-path mistakes that only failed together §6.4

ReadIndex records the commit index, confirms leadership with a heartbeat round, waits for the state machine to catch up, and only then answers. I got two things wrong at once:

The read index was captured when the request arrived, not when the leader actually knew what was committed. A node that has just restarted has commitIndex 0, and a read pinned to 0 requires nothing of the state machine at all.
Reads were resolved before entries were applied, so the answer was emitted ahead of the applies that would have brought the store up to date.

The trace was unambiguous: a node restarted at tick 9,040, answered a read nil at 10,099 with read_index = 0, and replayed its log afterwards in the same tick.

Every Raft invariant held. Only the linearizability checker saw it — which is the entire argument for having one. Re-introducing either mistake alone finds nothing across 3,000 seeds; it takes both.

Reproduce it live → press play, then "check linearizability" around tick 4,000 — the invariant panel stays green throughout

4. A rule of §6 I had skipped

The first end-to-end membership change never settled. Every seed showed the same shape: a server removed by the change, still holding only the joint entry — which includes it — no longer receiving heartbeats, timing out forever. Every RequestVote it sent carried a higher term and deposed a perfectly healthy leader. Term 178, commit index stuck at 6.

§6 describes this and prescribes the fix: a server that has heard from a leader within the minimum election timeout ignores vote requests entirely. I had implemented joint consensus and skipped the paragraph after it.

Worth noting why it went unnoticed for so long: without membership changes this looks like an ordinary partition, and I had already observed, tested and written it up as a known limitation of the base algorithm. Membership changes turn the same situation into a routine consequence of a normal operation.

5. §8's session table assumes one request in flight per client

With client retries switched on, the checker reported something that cannot be an ordering problem at all: put(k0, v23) = cas-failed(actual v20). A put that answered cas-failed.

The session table keeps a single slot per client, so with several requests in flight an old retry can arrive after a newer one has claimed the slot and be handed a completely different command's answer. The paper assumes clients send commands one at a time; my driver did not.

The checker now shape-checks results before searching — a Get must answer Value, a Put must answer Ok — so an answer belonging to a different command is reported as exactly that, instead of as a twenty-operation "no ordering exists".

Bugs it found in the checker

Three of the invariant checks were wrong, and all three were found by running at a scale the handcrafted tests never reach. The pattern is consistent enough to name: the checkers are hardest to get right where Raft's guarantees are conditional. Each of these was a property that holds only under a qualification, and the first draft asserted the unqualified version.

"A committed entry is never replaced", checked per node. A lagging follower is entitled to hold a stale entry at an index a majority has already committed — it simply has not been caught up yet. Replaced with a sound rule: a node must not truncate at or below its own commit index.
Leader Completeness without a term filter. It blamed a leader of term 12 for lacking an entry from term 13, which did not exist when that leader was elected.
Leader Completeness comparing the wrong term. §5.4 is about the term an entry was committed in, not created in — and because §5.4.2 exists precisely so earlier-term entries get committed later, those two are routinely far apart. A leader of term 37 was blamed for entries another node committed in term 38.

Checker validation runs in two directions. Planting bugs establishes that the checkers fire on real failures; it took 50,000 seeds to establish that they do not fire on correct behaviour. A checker with a noise floor is worse than no checker, because it trains you to ignore it.

Can it find a bug on purpose?

A harness reporting "no violations" is worth nothing unless it would have said otherwise. Every switch below is a real, historically common Raft mistake, run over 2,000 seeds unless noted.

bug	seeds that failed	caught by
ignore `votedFor` (§5.2)	1,857 / 2,000	invariants
uncapped `leaderCommit` (§5.3)	953 / 2,000	invariants
never persist hard state (§5.1)	385 / 2,000	invariants
compaction anchor at 0 (§7)	42 / 2,000	invariants
serve reads with no ReadIndex round (§6.4)	19 / 2,000	linearizability only
snapshot/log not reconciled (§7)	1 / 2,000	invariants
read index captured on arrival (§6.4)	1 / 2,000	linearizability only
commit previous-term entries (§5.4.2)	0 / 30,000	invariants

The last row is the honest one. The §5.4.2 violation — Figure 8, the most commonly botched rule in Raft — was found once in 20,000 seeds earlier in development and has not been reproduced since the search space widened. It needs a previous-term entry to reach a majority while the leader's own-term entry does not, and then a node holding a higher-term entry at that index to survive and win an election. That window is extremely narrow.

The handcrafted Figure 8 scenario finds it every single time, in milliseconds. That is the argument for keeping precise scenario tests alongside random search rather than treating the fuzzer as a replacement: random search covers the space you did not think of, and handcrafted tests cover the space you did.

One more lesson from this table. The first version of the harness could not find half the bugs it was given, because max_entries_per_append was pinned at 64: with a large batch a leader's backfill and its own-term entry always travel in the same message, so they commit together and §5.4.2 can never be violated. A configuration constant left fixed is a whole region of the search space that does not exist.