Test Gap Analyzer

Test Gap Analyzer — Quality Intelligence Engine

Automated coverage gap detection that feeds the Quality department's UAOP loop. Find what's untested, rank by risk, generate tests for the highest-risk gaps first. This is UAOP Stage 1 (Intelligence) + Stage 5 (Self-Improvement) for Quality.

When to use

• Before a release — "are we covered?"
• After adding new features — "what's untested?"
• After a production bug — "why didn't our tests catch this?"
• Sprint planning — "what should we test next?"
• Monthly quality review — "where are we improving, where are we regressing?"

When NOT to use

• Writing individual tests (use test-driven-development)
• Debugging a specific failure (use debug)
• Reviewing test methodology (use testing-methodology)

Pipeline

Step 1: Coverage Scan

Run coverage tools and collect raw data:

# Python coverage
python -m pytest tests/ --cov=src/coremind --cov-report=json --cov-report=term-missing -q

# Frontend coverage
cd frontend && npx vitest run --coverage

# E2E coverage (count vs. critical user flows)
# Compare existing E2E tests against documented user flows

Extract:

• Line coverage % — per module, per file
• Branch coverage % — conditional paths not exercised
• Uncovered functions — public functions with zero test coverage
• Uncovered files — entire files with no tests

Step 2: Risk-Rank Gaps

Not all gaps are equal. Rank by bug risk:

RISK SCORE = (Complexity × Change Frequency × User Impact) / Existing Coverage

Where:
  Complexity:       High (10) = many branches, external calls, state mutation
                    Medium (5) = moderate logic
                    Low (1) = simple getters, constants

  Change Frequency: Count of git commits touching this file in last 90 days

  User Impact:      Critical (10) = auth, payments, data mutation, API endpoints
                    High (7) = core business logic, agent execution
                    Medium (4) = internal utilities, helpers
                    Low (1) = dev tools, scripts, docs

  Existing Coverage: 0-100% (inverted — lower coverage = higher priority)

Step 3: Classify Gap Types

Step 4: Produce Gap Report

# Test Gap Report — [Date]

## Coverage Summary
| Suite | Current | Target | Delta | Status |
|-------|---------|--------|-------|--------|
| Python (pytest) | [X]% | 80% | [+/-Y]% | [OK/GAP] |
| Frontend (Vitest) | [X]% | 80% | [+/-Y]% | [OK/GAP] |
| E2E (Playwright) | [X/Y flows] | all critical | [N missing] | [OK/GAP] |
| API contracts | [X/Y endpoints] | all public | [N missing] | [OK/GAP] |

## Top 10 Gaps by Risk Score

| Rank | File/Function | Gap Type | Risk Score | Coverage | Recommendation |
|------|--------------|----------|------------|----------|---------------|
| 1 | [path:function] | [type] | [score] | [%] | [test type + approach] |

## Regression Debt
[List of production bugs without regression tests]

## Flaky Test Report
[List of flaky tests with failure rate and last occurrence]

## Improvement Trend
| Metric | Last Month | This Month | Delta |
|--------|-----------|------------|-------|
| Line coverage | [X]% | [Y]% | [+/-Z]% |
| Branch coverage | [X]% | [Y]% | [+/-Z]% |
| Bug escape rate | [X] | [Y] | [+/-Z] |
| Flaky test count | [X] | [Y] | [+/-Z] |

Step 5: Generate Test Targets

For each P0 gap, produce a test specification:

## Test Target: [File:Function]

**Gap type:** [Critical path / High-change / Branch / etc.]
**Risk score:** [X]
**Current coverage:** [X]%
**Why it matters:** [What could go wrong if this isn't tested]

### Tests to Write
1. `test_[function]_happy_path` — [expected behavior]
2. `test_[function]_returns_error_when_[condition]` — [error handling]
3. `test_[function]_handles_[edge_case]` — [boundary condition]
4. `test_[function]_with_[invalid_input]` — [input validation]

### Test Approach
- Pattern: [AAA / Property-based / Contract / E2E]
- Dependencies: [Real DB / Mock external / Fixture]
- Estimated effort: [S/M/L]

Step 6: Self-Improvement Loop (UAOP Stage 5)

After tests are written and run:

MEASURE:
  - Did new tests catch any existing bugs? (validation)
  - What's the coverage delta? (progress)
  - Any new flaky tests introduced? (quality)
  - Bug escape rate trend (effectiveness)

REINFORCE:
  - Test patterns that caught real bugs → prioritize more of these
  - Files that keep needing more tests → flag for refactoring
  - Coverage plateaus → switch from unit to property-based/mutation testing

REGENERATE:
  - Update gap report with new coverage data
  - Re-rank remaining gaps
  - Feed into next sprint's test targets

Cadence

WEEKLY:
  Monday:  Run coverage scan + produce gap report
  Tuesday: Prioritize top 5 gaps for the sprint
  Sprint:  Write tests targeting gaps
  Friday:  Re-scan, measure improvement, feed into next week

MONTHLY:
  Trend analysis — are we improving?
  Flaky test cleanup sprint
  Regression debt review

Tools Used

Agents

UAOP Integration

Stage 1 (Intelligence):  Coverage scan → gap identification → risk ranking
Stage 2 (Context):       Load quality/domain.md + quality/rules.md before any test writing
Stage 3 (Generation):    Produce test targets → write tests in parallel
Stage 4 (Split):         Auto-generated unit tests (T0) vs. human-designed E2E (T1)
Stage 5 (Loop):          Measure coverage delta → reinforce patterns → regenerate targets

Quality KPIs (Self-Improvement Metrics)