# Test Audit

> Zero-assumption test-suite audit focused on real confidence rather than coverage percentage. Use when reviewing test directories, individual suites, CI behavior, flaky tests, mock-heavy tests, or an application's full verification strategy to find false confidence and uncovered critical paths.

- Skill: `blakeox/test-audit` (Agent Skill)
- Install (CLI): `npx skillmds@latest add blakeox/test-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/blakeox/test-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: blakeox (https://skillmd.com/u/blakeox)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/blakeox/test-audit

---


Read `../_house-style/house-style.md` and `../_house-style/active-testing.md` before running the suite or setup commands.

## Anchor phrases

- 90% coverage with 0% edge case coverage is 0% useful coverage.
- A green test suite that doesn't cover failure modes is a confidence trick.
- Tests that break on every refactor test the implementation, not the behavior. They're maintenance cost disguised as safety.
- The most dangerous test is the one that passes when the code is wrong.
- "We have tests" is not the same as "we have confidence." Prove the difference.

## What you're measuring

You are not measuring coverage percentage. You are measuring **real confidence** — the probability that a bug in production would have been caught by this test suite before shipping.

A suite with 40% coverage that tests every error path in the payment service provides more confidence than a suite with 95% coverage that only tests happy paths and trivial getters.

## Audit process

### 1. Inventory the test suite

Before evaluating quality, understand the shape:

- **Test runner and framework** — what's configured, what version
- **Test count** — total, by type (unit, integration, e2e)
- **Test location pattern** — co-located, separate directory, both
- **Run time** — total suite time, slowest tests
- **Flaky tests** — any known or suspected
- **Coverage config** — is it measured? Thresholds? Enforced in CI?

### 2. Map tests to risk

For each major module/service in the codebase:

| Module | Risk level | Test count | Test quality | Confidence |
|---|---|---|---|---|
| Payment service | Critical | X | (below) | High/Med/Low |
| Auth middleware | Critical | X | | |
| User API | High | X | | |
| Utils | Low | X | | |

**Risk-weighted coverage** matters more than raw coverage. 0 tests on payment code is worse than 0 tests on a date formatter.

### 3. Classify every test file

For each test file, classify:

**Valuable tests** — test real behavior, would catch real bugs, survive refactors.

**Decoration tests** — exist for coverage numbers but catch nothing. Signs:
- Only test the happy path
- Assert that a function "was called" but not what it did
- Assert on implementation details (internal state, method call order)
- Test framework behavior, not application behavior
- Snapshot tests with no human review process

**False confidence tests** — worse than no tests because they make people think code is verified. Signs:
- Mock everything including the thing being tested
- Assert on mock return values (testing your setup, not your code)
- Pass even when the underlying code is broken
- Catch-all assertions (`expect(result).toBeTruthy()`)

**Maintenance burden tests** — break on every refactor, slow the suite, add friction. Signs:
- Tightly coupled to implementation details
- Brittle selectors/locators in UI tests
- Slow (>5s per test) without justification
- Duplicate existing test coverage

**Tests that should be deleted** — actively harmful. Removing them improves the suite.

### 4. Identify critical gaps

The most important part of the audit. What's NOT tested that SHOULD be:

- **Error paths** — what happens when things fail? Network errors, validation failures, auth rejection, timeout, malformed data
- **Edge cases** — empty inputs, boundary values, concurrent access, race conditions
- **Security paths** — auth bypass, privilege escalation, injection, IDOR
- **Data integrity** — what happens on partial failure, duplicate submission, stale reads
- **Integration seams** — where modules meet, where data changes shape, where trust boundaries exist

### 5. Evaluate test infrastructure

- **Factories/fixtures** — do they exist? Are they maintained? Do they reflect real data or fantasy data?
- **Test database** — real or mocked? If mocked, what's the divergence risk?
- **CI integration** — do tests run on every PR? Is there a quality gate? Can tests be skipped?
- **Flakiness** — is there a flaky test tracker? What's the rerun rate? Flaky tests erode trust in the entire suite.

## Per-test-file evaluation

For each test file worth examining:

- **What it claims to test** (from describe/context blocks)
- **What it actually tests** (from assertions)
- **What it misses** (paths not covered)
- **Mock health** — are mocks accurate? Do they drift from real implementations?
- **Assertion quality** — specific or vague? Behavior or implementation?
- **Verdict:** Valuable / Decoration / False confidence / Delete

## Output format

### Test suite health

| Metric | Value |
|---|---|
| Total tests | X |
| Valuable tests | X (Y%) |
| Decoration tests | X (Y%) |
| False confidence tests | X (Y%) |
| Tests to delete | X (Y%) |
| Suite run time | Xs |
| Flaky tests | X known |
| Real confidence level | High / Medium / Low / False |

### Risk-weighted coverage

| Module | Risk | Tests | Quality | Confidence | Verdict |
|---|---|---|---|---|---|
| (name) | Critical/High/Med/Low | count | Valuable/Mixed/Decoration | High/Med/Low | (action) |

### Critical gaps

The paths with no test coverage that have the highest blast radius. Each gap includes:

- **What's untested** — specific behavior/path
- **Why it matters** — what breaks if this path has a bug
- **Blast radius** — function → module → service → user-facing
- **Suggested test** — one-line description of the test that should exist

### False confidence report

Tests that are actively misleading. Each includes:

- **File:line** — the test
- **What it claims to verify** — from the test name/description
- **Why it doesn't** — what makes it a false positive
- **Verdict:** Fix or Delete

### Tests to delete

Specific tests that make the suite worse by existing. File:line, why they should go.

### Test infrastructure assessment

Factories, fixtures, mocking patterns, CI integration, flakiness. What's working, what's broken.

### Devil's advocate

For your harshest verdicts: could there be context you're missing? Were these tests valuable once and rotted? Is the team aware and choosing to accept the debt?

### What I didn't check

Types of testing I couldn't evaluate (e2e, performance, visual regression, manual QA processes).

### Recommended test plan

Priority-ordered list of tests to write, grouped by:

1. **Stop the bleeding** — tests for critical untested paths
2. **Build confidence** — tests that replace false confidence with real confidence
3. **Clean up** — tests to delete, mocks to replace, flakiness to fix