tokyubevoxelverse Detox
A flaky test is worse than a missing test: it trains everyone to ignore red. Your job is to separate flaky from broken, diagnose each flake's root cause class, and fix the cause — never the symptom.
Banned fixes
Retry wrappers, longer sleeps, and skip are not fixes; they are how flakes become permanent residents. The only acceptable retry is around a call to a genuinely external system the test cannot control — and that usually means the test should be mocking it instead.
Phase 1 — Census
Run the full suite repeatedly — default 5 runs, more if runs are cheap — and build a per-test pass/fail matrix.
- Fails every run → broken, not flaky. Report separately; out of scope unless asked.
- Fails some runs → flaky. This is the target list, ranked by failure rate.
- Record conditions per run: ordering, parallelism, seed, timing — the variance between a passing and failing run often names the culprit by itself.
Phase 2 — Diagnose each flake
Isolate, then classify. Standard experiments:
| Experiment | If behavior changes → |
|---|---|
| Run the test alone vs. after the full suite | Shared state / test-order dependence |
| Run the suite in random order / different parallelism | Order dependence, resource contention (ports, files, DB rows) |
| Run with a fixed seed / frozen clock | Unseeded randomness, time-of-day or timezone dependence |
| Run repeatedly in a tight loop | Race conditions, timing assumptions, hardcoded timeouts |
| Run with network blocked / observed | Real network calls hiding in a "unit" test |
Read the failing assertion under each hypothesis: a flake almost always belongs to one of six families — shared state, ordering, timing/races, unseeded randomness, clock/timezone, external systems (network, filesystem, ports).
Phase 3 — Fix the cause
Match the fix to the family: proper setup/teardown isolation for shared state; explicit awaiting or synchronization points instead of sleeps for races; injected/frozen clocks; seeded randomness; mocks or test doubles for external systems; per-test unique resources (temp dirs, ephemeral ports) for contention. The fix should make the test deterministic, not tolerant.
Phase 4 — Prove it
A fix isn't proven by one green run. Run the fixed test 20+ times, including under the conditions that made it fail (full-suite ordering, parallelism, tight loop). Then run the whole suite to confirm the fix didn't shift the flake elsewhere.
Phase 5 — Report
Write FLAKES.md: per test — failure rate before, root-cause family, the experiment that proved it, the fix, runs-passed after. Plus a short "flake debt" section: patterns in the codebase likely to breed the same families again (a shared fixture, a real HTTP client in unit tests), so the next flake gets prevented rather than hunted.