# Flaky Test Triage

> Use when a test fails intermittently or only on CI. Decide quarantine vs fix, then stabilize with determinism (fake clocks, isolation, retries at the right layer) — never by sprinkling sleeps.

- Skill: `magia187/flaky-test-triage` (Agent Skill)
- Install (CLI): `npx skillmds@latest add magia187/flaky-test-triage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/magia187/flaky-test-triage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: magia187 (https://skillmd.com/u/magia187)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/magia187/flaky-test-triage

---


# Flaky Test Triage

## When to use

- A test fails sometimes locally or on CI and passes on retry.
- CI is red due to known intermittent failures blocking merges.
- The user asks to "fix the flake", "quarantine", or "stabilize CI".

## When not to use

- Deterministic failures that reproduce every run (use repro-first).
- Production incidents unrelated to the test suite (use incident-hotfix).
- Rewriting the entire test framework as the first response.

## Assumptions

- Access to CI logs, test runner, and ability to re-run targeted tests.
- Quarantine mechanisms may exist (skip annotations, quarantine lists, issue links).
- Do **not** disable large swaths of the suite or delete tests without confirmation.
- Do **not** merge with `--no-verify` to bypass red CI without explicit approval.

## Workflow

1. Confirm flakiness: fail rate, environments, and whether order/parallelism matters.
2. Decide: fix now (default if bisectable) vs quarantine with a tracked issue (if blocking many engineers).
3. Root-cause toward non-determinism: time, randomness, shared state, network, ordering.
4. Fix with fakes/isolation/idempotent setup — not `sleep`.
5. Prove stability with repeated runs; remove quarantine when green.

## Steps

1. **Characterize** — Capture failing assertion, seed, CI node, and whether `--repeat` / reruns flake.
2. **Isolate** — Run the single test file repeatedly; try serial vs parallel to detect crosstalk.
3. **Decide** — Fix-now if cause is clear within a tight time box; else quarantine **one** test with owner + issue, not the whole file unless confirmed.
4. **Determinism toolkit** — Fake timers/clocks; seed RNGs; unique temp dirs; wait on conditions/events, not fixed sleeps; mock network; reset DB/fixtures per test.
5. **Guardrails** — Prefer explicit readiness checks (`waitFor`, polling with timeout) over arbitrary delay.
6. **Verify** — Re-run enough iterations to be confident; document the root cause in the PR/issue.

## Success criteria

- [ ] Flake is classified (time/random/shared state/network/order) or honestly unknown.
- [ ] Either a real fix lands, or a narrow quarantine with tracking issue and owner.
- [ ] No new fixed `sleep`/`setTimeout` delays used as the primary "fix".
- [ ] Repeated runs pass (or quarantine is verified to unblock CI intentionally).
- [ ] Broad test deletions or suite-wide skips were confirmed or avoided.

## Out of scope

- Permanent lowering of coverage goals to hide flakes.
- Infrastructure replacement (new CI vendor) as the first step.
- Load testing production from the test suite.