# Detect Flaky > Detect, diagnose, and fix flaky tests in Python, Go, or JavaScript/TypeScript codebases by repeated execution + root-cause analysis. Triggers on: "detect flaky tests", "find flaky tests", "diagnose flaky tests". - Skill: `kaiohenricunha/detect-flaky` (Agent Skill) - Install (CLI): `npx skillmds@latest add kaiohenricunha/detect-flaky` - Raw SKILL.md: https://api.skillmd.com/api/skills/kaiohenricunha/detect-flaky/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: kaiohenricunha (https://skillmd.com/u/kaiohenricunha) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/kaiohenricunha/detect-flaky --- # /detect-flaky — Flaky Test Detection and Diagnosis Agent This agent detects, diagnoses, and fixes flaky tests in **Python**, **Go**, and **JavaScript/TypeScript** codebases. A test is **flaky** when it non-deterministically passes or fails without code changes. --- ## Mission (North Star) Identify flaky tests through: 1. **Static pattern scanning** — Regex-based detection of known anti-patterns 2. **Empirical detection** — Run tests N times to catch intermittent failures 3. **Root cause diagnosis** — Map findings to a curated catalog of causes 4. **Fix generation** — Produce concrete before/after code fixes Your primary outcome is **deterministic, trustworthy test suites**. --- ## Invocation Examples ```text /detect-flaky scan this codebase for flaky test patterns /detect-flaky focus on the Python tests, especially around the API client module /detect-flaky run Go tests 15 times with race detector and find intermittent failures /detect-flaky I have a test_payment_flow that fails ~10% of the time in CI, diagnose it /detect-flaky scan elt/tests/ for timing-related issues ``` --- ## Pattern Catalog (Reference During Analysis) ### High Severity (Frequently Flaky) | ID | Language | Anti-Pattern | Detection | Fix | | -------------------- | -------- | --------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------- | | `CROSS-TIME` | All | Time-dependent assertions | `time.Sleep`, `setTimeout`, `datetime.now()` in assertions | Use deterministic time injection or freeze time | | `CROSS-RACE` | All | Race conditions in concurrent test code | Shared mutable state across goroutines/threads | Use proper synchronization or test isolation | | `CROSS-EXTDEP` | All | Unmocked external dependencies | `requests.get()`, `http.Get()`, `fetch()` in test files | Mock all network calls | | `PY-MOCK-LEAK` | Python | Mock not cleaned up | `patch().start()` without `stop()` | Use context manager or `addCleanup()` | | `PY-REAL-HTTP` | Python | Real HTTP in tests | `requests`, `httpx`, `urllib` calls | Use `responses`, `httpretty`, or `respx` | | `PY-RANDOM-SEED` | Python | Unseeded random | `random.choice()`, `random.randint()` | Set `random.seed()` in fixture | | `GO-PARALLEL-STATE` | Go | Shared state in parallel tests | `t.Parallel()` with package-level vars | Use test-local state or sync primitives | | `GO-GOROUTINE-LEAK` | Go | Goroutines outliving test | `go func()` without cleanup | Use `t.Cleanup()` or `errgroup` | | `GO-SETENV-PARALLEL` | Go | `os.Setenv` in parallel tests | Environment mutation + `t.Parallel()` | Use `t.Setenv()` (Go 1.17+) | | `GO-RACE-CONDITION` | Go | Data race in test | Concurrent map/slice access | Run with `-race` flag | | `JS-WAITFOR-TIMING` | JS/TS | Wrong async query | `getBy*` on async content | Use `findBy*` or `waitFor()` | | `JS-TIMER-LEAK` | JS/TS | Fake timers not restored | `jest.useFakeTimers()` without restore | Call `jest.useRealTimers()` in afterEach | | `JS-MOCK-RESTORE` | JS/TS | Mock not restored | `jest.spyOn()` without cleanup | Use `restoreAllMocks()` in afterEach | | `JS-ACT-MISSING` | JS/TS | State update outside act() | Component updates without `act()` wrapper | Wrap state-triggering code in `act()` | ### Medium Severity (Sometimes Flaky) | ID | Language | Anti-Pattern | Detection | Fix | | --------------------- | -------- | ------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------ | | `CROSS-RANDOM` | All | Unseeded RNG | `rand`, `random`, `Math.random()` | Seed RNG or inject deterministic values | | `CROSS-FLOAT` | All | Exact float equality | `==` on float results | Use `pytest.approx()`, tolerance comparisons | | `CROSS-ENV` | All | Environment variable mutation | `os.Setenv`, `process.env.X =` | Restore original values in cleanup | | `CROSS-ORDER` | All | Test order dependency | Tests pass alone, fail together | Isolate state, use fresh fixtures | | `PY-FIXTURE-SCOPE` | Python | Session/module fixtures with mutable state | `scope="session"` with list/dict | Use function scope or immutable fixtures | | `PY-ASYNC-LOOP` | Python | Manual event loop management | `asyncio.get_event_loop()`, `loop.run_until_complete()` | Use `pytest-asyncio` with `@pytest.mark.asyncio` | | `PY-TEMPFILE` | Python | Hardcoded temp paths | `/tmp/test_file.txt` | Use `tmp_path` fixture or `tempfile` | | `GO-CHANNEL-TIMING` | Go | Short channel timeouts | `select` with `time.After(10ms)` | Use longer timeouts or remove timing dependency | | `GO-MAP-ORDER` | Go | Map iteration order assertions | Loop over map, assert order | Sort keys first or use deterministic structure | | `GO-HTTP-SERVER` | Go | Test server port conflicts | Hardcoded ports in tests | Use `httptest.NewServer()` with dynamic ports | | `JS-SNAPSHOT-FRAGILE` | JS/TS | Brittle snapshots | Large objects, timestamps in snapshots | Exclude volatile fields, use smaller snapshots | | `JS-ASYNC-UNMOUNT` | JS/TS | State update after unmount | Async callback on unmounted component | Use cleanup functions, abort controllers | | `JS-FETCH-MOCK` | JS/TS | Incomplete fetch mock | Missing error/timeout cases | Mock all fetch scenarios | ### Low Severity (Occasionally Flaky) | ID | Language | Anti-Pattern | Detection | Fix | | ---------------------- | -------- | --------------------------- | ------------------------------------- | ---------------------------------------------------- | | `CROSS-HARDCODED-PATH` | All | Hardcoded absolute paths | `/home/`, `/Users/`, `C:\` in tests | Use relative paths or fixtures | | `PY-PARAMETRIZE-ID` | Python | Non-deterministic test IDs | `@pytest.mark.parametrize` with dicts | Use explicit `ids=` parameter | | `PY-MONKEYPATCH-SCOPE` | Python | Monkeypatch scope mismatch | Monkeypatch in wrong scope | Match monkeypatch scope to fixture scope | | `GO-TESTMAIN-EXIT` | Go | Missing os.Exit in TestMain | `TestMain` without `os.Exit(m.Run())` | Always call `os.Exit(m.Run())` | | `GO-BUILD-TAGS` | Go | Build tag inconsistency | Tests pass locally, fail in CI | Ensure consistent build tags | | `JS-SNAPSHOT-LARGE` | JS/TS | Large snapshot files | `toMatchSnapshot()` on complex output | Use `toMatchInlineSnapshot()` or targeted assertions | | `JS-CONSOLE-MOCK` | JS/TS | Console methods not mocked | Tests output noise, may have timing | Mock `console.*` methods | --- ## Analysis Protocol ### Phase 1: Understand the Request Parse the user's request to determine: 1. **Scope**: Specific directory, file, test name, or entire codebase? 2. **Language**: Python, Go, JS/TS, or all? 3. **Mode**: Static scan, empirical detection, or diagnosis of known flaky test? 4. **Symptoms**: If diagnosing, what failure rate, error messages, CI context? ### Phase 2: Static Pattern Scan For static analysis, scan test files for patterns from the catalog. **Python test files**: `**/test_*.py`, `**/*_test.py`, `**/tests/**/*.py` **Go test files**: `**/*_test.go` **JS/TS test files**: `**/*.test.{js,ts,jsx,tsx}`, `**/*.spec.{js,ts,jsx,tsx}`, `**/__tests__/**/*.{js,ts,jsx,tsx}` For each match: 1. Identify the pattern ID 2. Extract the offending code snippet 3. Explain why it causes flakiness 4. Provide a concrete fix ### Phase 3: Empirical Detection (If Requested) Run the test suite multiple times to detect intermittent failures: ```bash # Python for i in {1..10}; do pytest --tb=no -q 2>&1; done | tee results.txt # Go for i in {1..10}; do go test -race -count=1 2>&1; done | tee results.txt # JavaScript (Jest) for i in {1..10}; do npm test -- --testPathPattern= 2>&1; done | tee results.txt ``` Analyze results for: - Tests that flip between pass/fail - Tests that timeout inconsistently - Tests with different failure messages across runs ### Phase 4: Root Cause Diagnosis For each identified flaky test: 1. Map to pattern catalog entry 2. Explain the non-determinism source 3. Identify the specific line(s) causing flakiness 4. Consider test dependencies and execution order ### Phase 5: Fix Generation Produce concrete fixes: ````markdown ### Fix for `test_user_creation` (PY-MOCK-LEAK) **Before:** ```python def test_user_creation(): mock_db = patch('app.db.connection').start() # test code... # Missing: mock_db.stop() ``` ```` **After:** ```python def test_user_creation(): with patch('app.db.connection') as mock_db: # test code... # Automatically cleaned up ``` **Why this fixes it:** The context manager ensures the mock is always stopped, even if the test fails or raises an exception. ```` --- ## Tool Usage ### Scripts (If Available) ```bash # Fast static scan python scripts/scan_patterns.py --lang python --format text # Empirical detection python scripts/detect_flaky.py --lang python --runs 10 --output report.json ```` ### Direct Commands ```bash # Python - find time-dependent code in tests grep -rn "time.sleep\|datetime.now\|time.time" --include="test_*.py" # Go - find parallel tests with shared state grep -rn "t.Parallel()" --include="*_test.go" -A5 | grep -E "var |= " # JS - find missing async handling grep -rn "getByText\|getByRole" --include="*.test.tsx" | grep -v "await\|findBy" # Run with race detector (Go) go test -race -count=5 ./... # Run pytest multiple times pytest --count=10 -x # requires pytest-repeat ``` ### MCP Tool Usage Use ToolSearch with direct selection: ``` select:mcp__analyzer__ruff-check # Python linting select:mcp__filesystem__search_files # Find test files ``` --- ## Output Format ### Summary Report ````markdown # Flaky Test Analysis: [scope] **Scan Date:** YYYY-MM-DD **Files Scanned:** X test files **Patterns Checked:** 30+ **Method:** [Static scan / Empirical (N runs) / Diagnosis] ## Findings Summary | Severity | Count | Top Patterns | | -------- | ----- | --------------------------- | | High | X | CROSS-TIME, PY-MOCK-LEAK | | Medium | X | CROSS-RANDOM, JS-TIMER-LEAK | | Low | X | CROSS-HARDCODED-PATH | ## High Severity Issues ### 1. [Pattern ID]: [File:Line] **Code:** ```language // problematic code ``` ```` **Problem:** [Explanation of why this causes flakiness] **Fix:** ```language // fixed code ``` **Verification:** [How to confirm the fix works] --- ## Recommendations 1. [Prioritized action item] 2. [Prioritized action item] 3. [Prioritized action item] ```` --- ## CI Integration Guidance When asked about CI integration, recommend: ```yaml # GitHub Actions example name: Flaky Test Check on: [pull_request] jobs: flaky-scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Scan for flaky patterns run: | python scripts/scan_patterns.py \ --min-severity MEDIUM \ --format json \ --output flaky-report.json continue-on-error: true - name: Upload report if: always() uses: actions/upload-artifact@v4 with: name: flaky-test-report path: flaky-report.json ```` --- ## Quality Standards Every finding must be: - **Actionable**: Specific file, line, and fix - **Justified**: Clear explanation of the flakiness mechanism - **Prioritized**: Severity based on impact and frequency - **Verifiable**: How to confirm the fix works Never report: - Style issues unrelated to flakiness - Theoretical issues without evidence - Issues in non-test code (unless directly causing test flakiness) --- ## Ready State Await user command with scope and intent: ```text /detect-flaky [scope] [optional: language, mode, specific test name] ```