QA Flaky Detector

QA Flaky Detector

Purpose

Identify and classify flaky tests from CI history and test execution data. Flaky tests pass and fail intermittently for the same code, wasting CI time and masking real failures. This skill detects flakiness, classifies root causes into four patterns, and suggests targeted fixes.

Trigger Phrases

• "Find flaky tests" / "Identify flaky tests"
• "Analyze CI history for flakiness" / "Flaky test report"
• "Why does this test fail sometimes?" / "Intermittent test failures"
• "Classify flaky test [name]" / "Flaky test patterns"
• "Suggested fixes for flaky tests" / "Fix flaky [test name]"

4-Pattern Classification

See references/flaky-patterns.md for detailed patterns with code examples and fixes.

Detection Methods

1. CI History Analysis

• Same test, different results across runs (pass/fail flip)
• Compare JUnit XML, Allure, or CI job logs over N runs
• Threshold: pass rate < 100% over 10+ runs → flag as flaky

2. Statistical Analysis

• Pass rate per test: passes / (passes + failures) over recent runs
• Confidence: more runs → higher confidence in flakiness
• Minimum runs: recommend 10+ for reliable detection

3. Pattern Matching (Code-Based)

• Scan test code for known flaky signatures:
  • No await on async operations
  • setTimeout/setInterval without cleanup
  • new Date() or Date.now() without mocking
  • Direct HTTP calls, file I/O
  • Shared globals, singletons
  • Missing beforeEach/afterEach cleanup

See references/ci-analysis.md for CI history analysis methods.

Workflow

1. Input — CI artifacts (JUnit XML, Allure, GitHub Actions logs), test execution history, or test file paths
2. Collect — Parse results from N runs; build pass/fail matrix per test
3. Detect — Identify tests with pass rate < 100%
4. Classify — Map each flaky test to one or more of the 4 patterns (code scan + heuristics)
5. Suggest — Generate fix recommendations per pattern
6. Output — Flaky test report, prioritized fix list

Output Deliverables

Flaky Test Report

# Flaky Test Report — [Project/Branch]

## Summary
| Test | Failure Rate | Runs | Classification | Priority |
|------|--------------|------|----------------|----------|
| auth.login.spec.ts:42 | 23% | 26 | Race condition | High |
| checkout.flow.spec.ts:15 | 12% | 18 | Shared state | High |
| utils.date.spec.ts:8 | 8% | 12 | Time-dependency | Medium |

## Detailed Findings

### auth.login.spec.ts:42 — "should redirect after login"
- **Failure rate:** 6/26 (23%)
- **Classification:** Race condition
- **Likely cause:** DOM not ready before assertion
- **Suggested fix:** Add `await page.waitForSelector('#dashboard')` before assertion

Prioritized Fix List

1. High — High failure rate, critical path, or easy fix
2. Medium — Moderate rate, non-critical
3. Low — Low rate, edge cases

Integration with Other Skills

Scope

Can do (autonomous):

• Parse CI/test results (JUnit XML, Allure, common formats)
• Compute pass rate per test over N runs
• Classify flaky tests into 4 patterns
• Scan test code for flaky signatures
• Produce flaky report and prioritized fix list
• Suggest fixes per pattern (from reference patterns)

Cannot do (requires confirmation):

• Modify test code directly (suggest only; qa-test-healer can apply)
• Access private CI systems without credentials
• Override classification without evidence

Will not do (out of scope):

• Execute tests or run CI
• Deploy or change production
• Guarantee fix effectiveness (suggestions are heuristic-based)

Quality Checklist

• Pass rate calculated over sufficient runs (≥10 recommended)
• Each flaky test classified into at least one pattern
• Suggested fixes match pattern and reference examples
• Report includes test name, file:line, failure rate, classification
• Prioritized fix list ordered by impact
• No hardcoded credentials; CI access from user/env

Troubleshooting

Reference Files