# Fuzz Before You Claim Done > Before declaring a runtime-semantics change "done and verified", run a seeded property sweep over randomized inputs (graph shapes, schedules, interleavings) with invariants instead of expected outputs, plus instrumentation (TRAP logs) on the mechanism under test — because your hand-picked tests encode your own assumptions, and a design can be green on every one of them and still wrong. Two prior designs in the same arc were green on 1,638 tests and both double-fired. Make failures reproducible by seed alone. Use for schedulers, barriers, retry logic, caches, state machines — anything whose input space is combinatorial. Trigger terms: done and verified, all tests pass, edge cases covered, chaos test, property test, seeded fuzz, invariant, interleaving, crash test. - Skill: `serhiy-bzhezytskyy/fuzz-before-you-claim-done` (Agent Skill) - Install (CLI): `npx skillmds@latest add serhiy-bzhezytskyy/fuzz-before-you-claim-done` - Raw SKILL.md: https://api.skillmd.com/api/skills/serhiy-bzhezytskyy/fuzz-before-you-claim-done/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: serhiy-bzhezytskyy (https://skillmd.com/u/serhiy-bzhezytskyy) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/serhiy-bzhezytskyy/fuzz-before-you-claim-done --- # Fuzz before you claim "done" ## Purpose A hand-picked test list is a mirror: it verifies the cases its author imagined, and the author of the tests is usually the author of the change. In someone else's repo that gap lands on the maintainer — the reviewer becomes your fuzzer. A seeded property sweep is the cheapest adversarial reviewer you can hire before the human one: randomized structures, deterministic per seed, checked against *invariants* ("no writes lost", "exactly once", "resume equals uninterrupted") rather than expected outputs. When it fails, the seed is the whole repro. ## When to use - Any change to execution semantics: scheduling, barriers/joins, retries, caching, interrupts/resume, checkpointing — anything where inputs combine. - Before writing "verified" / "all cases covered" in a PR body or a handoff. - When your deterministic suite is green and you notice every test was written by the same mind that wrote the code. ## When NOT to use Pure data transforms with enumerable inputs — a table-driven test is clearer. And don't ship the 10,000-seed run in CI: sweep large once, commit a small seeded subset, record the large run's result where the reviewer can see it. Two adjacent adversaries answer different doubts, and a fuzz run answers neither: [red-team-your-own-diff](../red-team-your-own-diff/SKILL.md) attacks the diff with a named reviewer's own recorded rules, and [run-the-bots-review-before-it-does](../run-the-bots-review-before-it-does/SKILL.md) pre-runs the automated reviewer's knowable checklist. Volume in one of the three is not coverage of the other two. ## The practice (checklist) - [ ] **Generate structures, not just values**: random graph/schedule shapes — branches of different depths, loops with bounded exits, dynamic dispatch, 1–2 instances of the feature mixed with defaults. - [ ] **Deterministic per seed** (e.g. mulberry32): a failure message carrying only its seed must reproduce the run exactly. - [ ] **Assert invariants, not outputs**: "target runs exactly once iff a source ran", "a completed run leaves nothing armed", "interrupt+resume produces the uninterrupted result". Derive the oracle from the contract, not from the code. - [ ] **Instrument the mechanism (TRAP)**: log every state transition of the thing under test (each barrier write with before/after) — buffered per seed, dumped only on failure. A failing seed without a trace is a second investigation. - [ ] **Differential where possible**: run the same seed uninterrupted vs interrupted-and-resumed, or feature-on vs feature-off, and compare. - [ ] **Classify non-done outcomes** (recursion limit, error) instead of ignoring them — "limit hit on a loop-free graph" is a finding, not noise. - [ ] **Chase the literature's named pathologies as targeted cases** (for joins: the OR-join vicious circle) — fuzz finds the unknown unknowns, the literature names the known ones. - [ ] When a failure is a *semantics surprise* rather than a bug, **pin it as a deterministic test and document the behavior** — the fuzz just wrote your docs. - [ ] **Vary the axes the battery holds constant, not only the inputs**: any flag that changes a persisted shape gets a flip test — create state under configuration A, resume under configuration B, both directions. 2,400 seeds missed a silent thread-corruption bug because every seed compiled the graph once; the bug lived in recompiling with a different flag over a live thread. ## Rationalizations | Shortcut | Why it fails | |---|---| | "I have N passing tests, including the tricky cases." | Two designs in the same problem space were green on 1,638 tests each; both double-fired on a case no hand-picked list contained. The suite measured the author's imagination, not the design. | | "The feature is deterministic, there's nothing to fuzz." | The feature is; its *context* isn't. The real defect was an interrupt landing on one specific superstep boundary — a placement no one would hand-pick, and seeds 302/306/307 all found it in under a second. | | "Randomized tests are flaky." | Seeded PRNG + invariants = exactly as deterministic as any unit test, plus a one-integer repro. Flakiness comes from unseeded randomness and output-matching, not from the method. | | "I'll fuzz after review if they ask." | Then the reviewer is your fuzzer, at the most expensive point in the pipeline. The sweep costs seconds and its findings arrive before anyone's trust is spent. | | "A big fuzz run doesn't belong in the suite." | Correct — so commit the 300-seed version and record the 2,400-seed result in the PR/notes. Don't let CI cost cancel the practice. | ## RECEIPT *SOURCING-receipt, internal (pending publication of the branch it guarded).* 2026-08-14, `langgraphjs` inclusive-waiting-edge work: a 2,400-seed sweep over random graph shapes (conditional entry subsets, chains at different depths, bounded loops, `Send` dispatchers) with every barrier write TRAP-logged, run **after** 12 hand-picked tests were green on the first attempt. Round 1: 29 failing seeds → one real defect the hand-picked list could not have expressed — a run interrupted exactly at the release point reported the documented "run is over" signal (`next: []`), so a client following the docs never resumed and the release never happened — plus one undocumented semantics (once-per-arming on double-triggered sources) that became JSDoc and a pinned test. Round 2, after the fix: 0 of 2,400. The same day, targeted literature cases (OR-join vicious circle) and a kill -9 crash test rode the same harness. Evidence: archived probes with as-run results in the practice's private flight recorder; ripens into a public receipt when the branch ships. ## Lifecycle Sharpen after each use: record in the LEDGER which invariant caught something, which generated shape never fired (delete it), and every claim the sweep proved wrong. Retire if two consecutive uses on real work produce zero findings beyond the deterministic suite — that would mean the suites got better, which is the goal.