# Flaky Test Analyzer

> Use when tests pass locally but fail in CI, pass some runs and fail others, or someone suspects a test is unreliable but cannot reproduce the failure consistently. Diagnoses root causes and provides concrete fixes. Triggers on: "flaky test".

- Skill: `newkayak12/flaky-test-analyzer` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add newkayak12/flaky-test-analyzer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/newkayak12/flaky-test-analyzer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: newkayak12 (https://skillmd.com/u/newkayak12)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/newkayak12/flaky-test-analyzer

---


# Flaky Test Analyzer

## When to Use / When Not to Use

**Use when:**
- A test fails intermittently without code changes
- Tests pass locally but fail in CI
- Re-runs are being used as a workaround (treating symptoms, not the cause)

**Do not use when:**
- The test fails consistently — that is a bug, not flakiness
- You need to write new tests (use `test-master` or `test-driven-development`)

## Process

1. **Triage first** — identify the failure category before attempting a fix
2. **Reproduce in isolation** — run only the failing test 20 times
3. **Reproduce with ordering** — run the failing test after each other test in the suite
4. **Add logging** — capture timestamps, thread names, and state snapshots at failure time
5. **Check nondeterministic inputs** — search for `new Date()`, `Math.random()`, `UUID.randomUUID()`, `System.currentTimeMillis()`
6. **Check resource cleanup** — verify `@AfterEach` teardown, unclosed streams, un-stubbed mocks

If `sequential-thinking` is available, use it to work through steps 1–5 in order — skipping triage before fixing is the primary failure mode.

## Output Template

For each flaky test analysis, provide:
1. Failure category (from the triage table below)
2. Root cause diagnosis with evidence
3. Concrete fix (code snippet)
4. Prevention rule for the test suite
5. Recommended CI configuration to catch this class of flakiness earlier

## What Claude Does / What You Do

| Claude | You |
|--------|-----|
| Identifies failure category from symptom description | Provide the test code and failure log |
| Explains why the root cause causes intermittent failure | Run the test 20 times in isolation to confirm |
| Provides the specific fix pattern (waitFor, Clock injection, etc.) | Implement and verify the fix |
| Suggests `@BeforeEach` / `@AfterEach` cleanup patterns | Apply to the full test suite |
| Recommends CI flags (random ordering, etc.) | Configure CI pipeline |

## Triage: Find the Category First

| Fails When | Category |
|-----------|----------|
| Run repeatedly in isolation | Timing dependency or resource leak |
| Run after specific other tests | Ordering / shared state dependency |
| Run in parallel | Concurrency or shared resource conflict |
| Run on CI but not locally | Environment dependency (clock, timezone, path, env var) |
| Run with real external systems | External dependency (network, DB, third-party API) |
| Passes after a sleep/wait | Timing / async race condition |

## The 5 Flakiness Categories

**1. Timing Dependencies** — Asserts on async work before it completes.
Fix: use `waitFor`/`awaitility`, event-driven signals, or inject a controllable `Clock`. Never use `Thread.sleep`.

**2. Shared State Between Tests** — Tests pollute database, static variables, in-memory caches, or file system.
Fix: reset state in `@BeforeEach`/`@AfterEach`; use `@Transactional` rollback or explicit truncate; prefer instance injection over singletons.

**3. External Dependencies** — Tests call real HTTP APIs, databases, or message queues.
Fix: mock at the boundary (WireMock, Mockito); use Testcontainers for integration tests needing a real DB.

**4. Test Ordering Dependencies** — Test B implicitly relies on state created by Test A.
Fix: self-contained setup per test; run tests in random order (`--randomly-seed=random`).

**5. Concurrency and Parallelism** — Parallel tests share ports, files, or singletons.
Fix: use port 0 (OS-assigned); inject resources so each test gets its own instance.

## Fix Patterns Quick Reference

| Root Cause | Fix |
|-----------|-----|
| Async race condition | Use `waitFor` / `awaitility`; never `Thread.sleep` |
| System clock dependency | Inject `Clock`; use fixed clock in tests |
| Database pollution | `@Transactional` rollback or truncate in `@BeforeEach` |
| Static/singleton state | Reset in `@BeforeEach`/`@AfterEach`; prefer instance injection |
| Real HTTP/DB calls | Mock with WireMock, Mockito, or Testcontainers |
| Ordering dependency | Self-contained setup; run tests in random order |
| Port conflict | Use port 0; never hardcode test ports |
| Timezone sensitivity | Set `TZ=UTC` in CI; use `ZonedDateTime` not `Date` |
| Random data collisions | Use unique test data per run (UUID prefix) |

For category-specific fix code examples, see `references/flaky-fix-patterns.md`.

## Prevention

- Run tests in random order in CI by default
- Quarantine any test that flakes more than 1% of runs
- Never commit a "re-run on failure" workaround without a ticket to fix the root cause
- Fail the build on any `@Disabled` / `@Ignore` without a linked issue

## Related Skills

- `test-master` — writing new tests with built-in flakiness prevention
- `test-driven-development` — TDD patterns that reduce flakiness by design
- `chaos-engineer` — when you want to intentionally test failure behavior