newkayak12 Flaky Test Analyzer
When to Use / When Not to Use
Use when:
- A test fails intermittently without code changes
- Tests pass locally but fail in CI
- Re-runs are being used as a workaround (treating symptoms, not the cause)
Do not use when:
- The test fails consistently — that is a bug, not flakiness
- You need to write new tests (use
test-masterortest-driven-development)
Process
- Triage first — identify the failure category before attempting a fix
- Reproduce in isolation — run only the failing test 20 times
- Reproduce with ordering — run the failing test after each other test in the suite
- Add logging — capture timestamps, thread names, and state snapshots at failure time
- Check nondeterministic inputs — search for
new Date(),Math.random(),UUID.randomUUID(),System.currentTimeMillis() - Check resource cleanup — verify
@AfterEachteardown, unclosed streams, un-stubbed mocks
If sequential-thinking is available, use it to work through steps 1–5 in order — skipping triage before fixing is the primary failure mode.
Output Template
For each flaky test analysis, provide:
- Failure category (from the triage table below)
- Root cause diagnosis with evidence
- Concrete fix (code snippet)
- Prevention rule for the test suite
- Recommended CI configuration to catch this class of flakiness earlier
What Claude Does / What You Do
| Claude | You |
|---|---|
| Identifies failure category from symptom description | Provide the test code and failure log |
| Explains why the root cause causes intermittent failure | Run the test 20 times in isolation to confirm |
| Provides the specific fix pattern (waitFor, Clock injection, etc.) | Implement and verify the fix |
Suggests @BeforeEach / @AfterEach cleanup patterns |
Apply to the full test suite |
| Recommends CI flags (random ordering, etc.) | Configure CI pipeline |
Triage: Find the Category First
| Fails When | Category |
|---|---|
| Run repeatedly in isolation | Timing dependency or resource leak |
| Run after specific other tests | Ordering / shared state dependency |
| Run in parallel | Concurrency or shared resource conflict |
| Run on CI but not locally | Environment dependency (clock, timezone, path, env var) |
| Run with real external systems | External dependency (network, DB, third-party API) |
| Passes after a sleep/wait | Timing / async race condition |
The 5 Flakiness Categories
1. Timing Dependencies — Asserts on async work before it completes.
Fix: use waitFor/awaitility, event-driven signals, or inject a controllable Clock. Never use Thread.sleep.
2. Shared State Between Tests — Tests pollute database, static variables, in-memory caches, or file system.
Fix: reset state in @BeforeEach/@AfterEach; use @Transactional rollback or explicit truncate; prefer instance injection over singletons.
3. External Dependencies — Tests call real HTTP APIs, databases, or message queues. Fix: mock at the boundary (WireMock, Mockito); use Testcontainers for integration tests needing a real DB.
4. Test Ordering Dependencies — Test B implicitly relies on state created by Test A.
Fix: self-contained setup per test; run tests in random order (--randomly-seed=random).
5. Concurrency and Parallelism — Parallel tests share ports, files, or singletons. Fix: use port 0 (OS-assigned); inject resources so each test gets its own instance.
Fix Patterns Quick Reference
| Root Cause | Fix |
|---|---|
| Async race condition | Use waitFor / awaitility; never Thread.sleep |
| System clock dependency | Inject Clock; use fixed clock in tests |
| Database pollution | @Transactional rollback or truncate in @BeforeEach |
| Static/singleton state | Reset in @BeforeEach/@AfterEach; prefer instance injection |
| Real HTTP/DB calls | Mock with WireMock, Mockito, or Testcontainers |
| Ordering dependency | Self-contained setup; run tests in random order |
| Port conflict | Use port 0; never hardcode test ports |
| Timezone sensitivity | Set TZ=UTC in CI; use ZonedDateTime not Date |
| Random data collisions | Use unique test data per run (UUID prefix) |
For category-specific fix code examples, see references/flaky-fix-patterns.md.
Prevention
- Run tests in random order in CI by default
- Quarantine any test that flakes more than 1% of runs
- Never commit a "re-run on failure" workaround without a ticket to fix the root cause
- Fail the build on any
@Disabled/@Ignorewithout a linked issue
Related Skills
test-master— writing new tests with built-in flakiness preventiontest-driven-development— TDD patterns that reduce flakiness by designchaos-engineer— when you want to intentionally test failure behavior