Flaky Test Audit
Purpose
Restore trust in the test suite by eliminating flakiness at its source: identify tests that pass and fail without code changes, diagnose the real cause, and fix it — because a flaky suite gets ignored, then bypassed, then guards nothing.
When to Use
- When tests fail intermittently, CI needs reruns to go green, or people say "just re-run it."
- Proactively before flakiness normalizes bypassing.
- Not for genuinely-failing tests (those are real bugs —
../../bug-investigation).
Inputs
- CI history / rerun logs showing intermittent failures.
- The flaky tests + their environment (
test-environment-management) and data (test-data-management).
Discovery Questions
- Which tests fail without related code changes, and how often?
- Does the failure depend on order, parallelism, timing, or wall-clock/date?
- Does it touch shared state, real time/randomness, or a live external?
Responsibilities
- Detect: mine CI for pass/fail-without-change signals; rank by frequency and by what they guard.
- Diagnose the root cause class:
- timing (fixed sleeps, races, missing awaits) → deterministic waits / auto-wait (
playwright-e2e/maestro-e2e);
- shared/leaked state or order dependence → per-test isolation (
test-data-management);
- real clock/randomness → fake timers, seeded randomness;
- live externals / network → fakes at interfaces (
test-environment-management);
- environment nondeterminism (fonts, viewport, locale) → pin it (
visual-regression).
- Fix the cause, not the symptom: no blanket retries, no
sleep bumps, no .skip as the resolution.
- Quarantine sparingly and temporarily: a persistently-flaky test may be isolated from the gate with a tracked ticket and an owner, never silently disabled — and fixed or deleted promptly.
- Add guardrails: run tests in random order in CI to surface order dependence; treat new flakiness as a same-day issue.
Required Workflow
- Detect flaky tests from CI history; rank by impact.
- Reproduce (repeat runs, randomized order, parallelism).
- Diagnose the root-cause class.
- Fix the cause at the right layer.
- Quarantine only if unavoidable — ticketed, owned, time-boxed; verify stability post-fix.
Decision Rules
- Fix the root cause; retries and sleep-bumps hide flakiness and let it spread.
- Randomized-order CI runs are the cheapest way to expose order dependence — enable them.
- Quarantine is a temporary bridge with a ticket, not a resting place; a skipped test guards nothing.
- A flaky test that can't be made deterministic is a candidate for deletion, not permanent retry.
Rules
- No blanket auto-retries masking flakiness.
- Quarantined tests are tracked, owned, and time-boxed.
- New flakiness is treated as a real defect, not noise.
Anti-Patterns
- "Just re-run CI" as the standard response.
- Adding
waitForTimeout/sleeps to stabilize timing.
.skip as the permanent fix.- Global test retries hiding real races.
- Ignoring order dependence by always running in the same order.
Validation Checklist
Definition of Done
Flaky tests identified, root-caused, and fixed at the source — with randomized-order CI guarding against regression and any quarantine tracked and time-boxed — so the suite is trusted again.
Related Skills
test-data-management, test-environment-management, playwright-e2e, maestro-e2e, visual-regression, test-coverage-audit, ../../bug-investigation, ../../devops/ci-cd.
Related Knowledge
../../../knowledge/ (known flaky areas, CI history).
Related References
../../../references/testing/ (flakiness patterns, when populated).
Context Loading Guidance
- Requires: CI failure history, the flaky tests + their env/data.
- Does not require: the whole suite, unrelated passing tests.
- May load:
test-data-management, test-environment-management.
- Stop when: causes are fixed and the suite is stable under randomized order.
Token Efficiency Guidance
Rank by impact and work the top offenders; the flaky-test → cause → fix table is the artifact, not a full-suite review.
1---2name: flaky-test-audit3description: Use to find, diagnose, and fix flaky tests — nondeterministic passes/failures from timing, shared state, ordering, real time/randomness, or live externals. Fix the root cause; quarantine only temporarily. A trusted suite is the goal, not a green-by-retry one.4---56# Flaky Test Audit78## Purpose910Restore trust in the test suite by eliminating flakiness at its source: identify tests that pass and fail without code changes, diagnose the real cause, and fix it — because a flaky suite gets ignored, then bypassed, then guards nothing.1112## When to Use1314- When tests fail intermittently, CI needs reruns to go green, or people say "just re-run it."15- Proactively before flakiness normalizes bypassing.16- **Not** for genuinely-failing tests (those are real bugs — `../../bug-investigation`).1718## Inputs1920- CI history / rerun logs showing intermittent failures.21- The flaky tests + their environment (`test-environment-management`) and data (`test-data-management`).2223## Discovery Questions2425- Which tests fail without related code changes, and how often?26- Does the failure depend on order, parallelism, timing, or wall-clock/date?27- Does it touch shared state, real time/randomness, or a live external?2829## Responsibilities3031- **Detect**: mine CI for pass/fail-without-change signals; rank by frequency and by what they guard.32- **Diagnose the root cause class**:33 - **timing** (fixed sleeps, races, missing awaits) → deterministic waits / auto-wait (`playwright-e2e`/`maestro-e2e`);34 - **shared/leaked state** or **order dependence** → per-test isolation (`test-data-management`);35 - **real clock/randomness** → fake timers, seeded randomness;36 - **live externals / network** → fakes at interfaces (`test-environment-management`);37 - **environment nondeterminism** (fonts, viewport, locale) → pin it (`visual-regression`).38- **Fix the cause**, not the symptom: no blanket retries, no `sleep` bumps, no `.skip` as the resolution.39- **Quarantine sparingly and temporarily**: a persistently-flaky test may be isolated from the gate *with a tracked ticket and an owner*, never silently disabled — and fixed or deleted promptly.40- Add guardrails: run tests in random order in CI to surface order dependence; treat new flakiness as a same-day issue.4142## Required Workflow43441. Detect flaky tests from CI history; rank by impact.452. Reproduce (repeat runs, randomized order, parallelism).463. Diagnose the root-cause class.474. Fix the cause at the right layer.485. Quarantine only if unavoidable — ticketed, owned, time-boxed; verify stability post-fix.4950## Decision Rules5152- Fix the root cause; retries and sleep-bumps hide flakiness and let it spread.53- Randomized-order CI runs are the cheapest way to expose order dependence — enable them.54- Quarantine is a temporary bridge with a ticket, not a resting place; a skipped test guards nothing.55- A flaky test that can't be made deterministic is a candidate for deletion, not permanent retry.5657## Rules5859- No blanket auto-retries masking flakiness.60- Quarantined tests are tracked, owned, and time-boxed.61- New flakiness is treated as a real defect, not noise.6263## Anti-Patterns6465- "Just re-run CI" as the standard response.66- Adding `waitForTimeout`/sleeps to stabilize timing.67- `.skip` as the permanent fix.68- Global test retries hiding real races.69- Ignoring order dependence by always running in the same order.7071## Validation Checklist7273- [ ] Flaky tests detected + ranked from CI history.74- [ ] Root-cause class diagnosed per test.75- [ ] Cause fixed at the right layer (no retry/sleep bandaids).76- [ ] Any quarantine is ticketed, owned, time-boxed.77- [ ] Randomized-order runs enabled; stability verified.7879## Definition of Done8081Flaky tests identified, root-caused, and fixed at the source — with randomized-order CI guarding against regression and any quarantine tracked and time-boxed — so the suite is trusted again.8283## Related Skills8485`test-data-management`, `test-environment-management`, `playwright-e2e`, `maestro-e2e`, `visual-regression`, `test-coverage-audit`, `../../bug-investigation`, `../../devops/ci-cd`.8687## Related Knowledge8889`../../../knowledge/` (known flaky areas, CI history).9091## Related References9293`../../../references/testing/` (flakiness patterns, when populated).9495## Context Loading Guidance9697- **Requires:** CI failure history, the flaky tests + their env/data.98- **Does not require:** the whole suite, unrelated passing tests.99- **May load:** `test-data-management`, `test-environment-management`.100- **Stop when:** causes are fixed and the suite is stable under randomized order.101102## Token Efficiency Guidance103104Rank by impact and work the top offenders; the flaky-test → cause → fix table is the artifact, not a full-suite review.
ahtishamshahzad