Flaky Test Triage

Flaky Test Triage

When to use

• A test fails sometimes locally or on CI and passes on retry.
• CI is red due to known intermittent failures blocking merges.
• The user asks to "fix the flake", "quarantine", or "stabilize CI".

When not to use

• Deterministic failures that reproduce every run (use repro-first).
• Production incidents unrelated to the test suite (use incident-hotfix).
• Rewriting the entire test framework as the first response.

Assumptions

• Access to CI logs, test runner, and ability to re-run targeted tests.
• Quarantine mechanisms may exist (skip annotations, quarantine lists, issue links).
• Do not disable large swaths of the suite or delete tests without confirmation.
• Do not merge with --no-verify to bypass red CI without explicit approval.

Workflow

1. Confirm flakiness: fail rate, environments, and whether order/parallelism matters.
2. Decide: fix now (default if bisectable) vs quarantine with a tracked issue (if blocking many engineers).
3. Root-cause toward non-determinism: time, randomness, shared state, network, ordering.
4. Fix with fakes/isolation/idempotent setup — not sleep.
5. Prove stability with repeated runs; remove quarantine when green.

Steps

1. Characterize — Capture failing assertion, seed, CI node, and whether --repeat / reruns flake.
2. Isolate — Run the single test file repeatedly; try serial vs parallel to detect crosstalk.
3. Decide — Fix-now if cause is clear within a tight time box; else quarantine one test with owner + issue, not the whole file unless confirmed.
4. Determinism toolkit — Fake timers/clocks; seed RNGs; unique temp dirs; wait on conditions/events, not fixed sleeps; mock network; reset DB/fixtures per test.
5. Guardrails — Prefer explicit readiness checks (waitFor, polling with timeout) over arbitrary delay.
6. Verify — Re-run enough iterations to be confident; document the root cause in the PR/issue.

Success criteria

• Flake is classified (time/random/shared state/network/order) or honestly unknown.
• Either a real fix lands, or a narrow quarantine with tracking issue and owner.
• No new fixed sleep/setTimeout delays used as the primary "fix".
• Repeated runs pass (or quarantine is verified to unblock CI intentionally).
• Broad test deletions or suite-wide skips were confirmed or avoided.

Out of scope

• Permanent lowering of coverage goals to hide flakes.
• Infrastructure replacement (new CI vendor) as the first step.
• Load testing production from the test suite.