# Flaky Detector

> Detect flaky tests by running the test suite multiple times and comparing results.

- Skill: `llp42/flaky-detector` (Agent Skill)
- Install (CLI): `npx skillmds@latest add llp42/flaky-detector`
- Raw SKILL.md: https://api.skillmd.com/api/skills/llp42/flaky-detector/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: llp42 (https://skillmd.com/u/llp42)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/llp42/flaky-detector

---


Detect flaky tests by running the test suite multiple times and comparing results.

1. Determine the test command:
   - If `$1` is given, use it as the test command.
   - Otherwise auto-detect, in this order:
     - `package.json` has a `test` script -> use `npm test`
     - `pytest.ini` or `pyproject.toml` exists -> use `pytest`
     - `go.mod` exists -> use `go test ./...`
     - `Cargo.toml` exists -> use `cargo test`
   - If none found and no `$1` given, report that no test command could be detected and stop.

2. Determine number of runs: use `$2` if given as an integer, else default to 5.

3. Run the test command that many times in a row, one after another. Capture the full stdout/stderr of each run. Do NOT stop early if a run fails — always complete all configured runs.

4. For each run's output, parse individual test names and their pass/fail status, using best-effort text parsing appropriate to the detected framework's typical output (e.g. Jest/Mocha "✓ name" / "✗ name" or "PASS/FAIL", pytest "PASSED"/"FAILED" per test id, go test "--- PASS:"/"--- FAIL:" per test name, cargo test "test X ... ok/FAILED").

5. Aggregate results per test name across all runs:
   - If a test passed in all runs, or failed in all runs, it is NOT flaky.
   - If a test passed in some runs and failed in others, it IS flaky. Report it with its failure rate, e.g. "test_foo: 2/5 runs failed".

6. Output a final summary:
   - List each flaky test found with its failure rate.
   - If no flaky tests were found, state that plainly in one line (e.g. "No flaky tests detected across 5 runs.").