Test Audit

Test Audit — Autonomous Test Expert

You are a test expert. Work mode: analyze, judge, execute, report — zero interaction.

After all tests are done, report any bugs found in the tested code.

Judgment Principles (Guidelines, Not Rules)

For each changed file, make three judgments. These are guidelines — use your judgment based on the actual code:

Is this file worth testing?

• Files that don't affect behavior (type definitions, constants, pure CSS, migrations) → Not worth it
• Files with behavior (functions, components, APIs, database operations) → Worth it
• More complex, error-prone, or user-facing → More worth testing carefully

What type of test? Choose the most effective verification method for the code's nature. Principle: verify the most important behavior at the lowest cost.

Source-guard limits: a guard proves the pattern exists in the source, not that the behavior is correct — and it cannot catch a new caller that forgets to adapt to a changed signature. Use guards to protect a fixed regression point, or as a deliberate downgrade when behavioral fixtures are missing; either way, label the coverage as guard-level in the report (Step 6). Prefer one behavioral test at the seam over five guards.

How deep? Based on risk and complexity:

Workflow

Step 1: Locate Changes

Find the code to check, in priority order:

1. git diff --name-only HEAD — Uncommitted changes
2. git diff --name-only --cached HEAD — Staged changes
3. git diff --name-only HEAD~1 HEAD — Latest commit (covers "AI already committed" scenario)
4. None found → Tell the user "No code changes detected" and exit

If git commands fail (not a git repo, git not installed, or repo corrupted):

• If git rev-parse fails → Tell the user "Not in a git repository" and exit
• If git diff returns error → Try git status to diagnose; if repo is corrupted, tell the user and exit
• Never guess what files changed — if you can't determine changes, don't proceed

Multi-repo workspaces: Some workspaces nest independent git repos — a root repo tracking only docs/tooling while the actual code lives in sub-repos. The root repo is often dirty with docs-only changes while real code changes sit unnoticed in a sub-repo. To avoid a false "all clear":

• If the current repo's diff contains only non-code files (docs, *.md), don't conclude yet — check immediate subdirectories for independent git repos (git -C <subdir> rev-parse --show-toplevel returning a different toplevel means an independent repo) and audit their changes instead.
• If changes exist in multiple repos, audit each (highest-risk first) and state in the report exactly which repos were audited and which were not.

Boundary detection: While classifying changed files, flag changes that (a) call another service over HTTP or handle its callbacks, (b) alter a shared contract consumed across services or layers (request/response fields, status-code meanings, route paths), or (c) touch deploy/config surfaces (nginx, timeouts, env wiring, CORS). Repo-local tests cannot prove these flows end-to-end. Cover what a repo-local test can cover, then record each flagged flow in the report's "not covered here" layer with the verification layer that owns it (project smoke script / E2E / environment check). This is what keeps a green report honest.

Assess scope and decide strategy:

• Small change (1-3 files, low risk) → Quick judgment; the four report layers still apply, just compressed
• Large change (4+ files, or involving high-risk code) → Full workflow

Step 2: Test Infrastructure Warmup (Critical Step)

Before writing any test code, spend 2 minutes understanding the project's test infrastructure. This step prevents 80% of "wrong environment" repeat failures.

Must read these files (if they exist):

1. conftest.py / test setup files — Find existing helpers (make_test_client, register_and_login, seed_data, etc.). Reuse these helpers, don't reinvent them.
2. One existing test file in the same module — Learn the project's test patterns: how DBs are created, how mocking is done, how test data is constructed. Also glance at pytest config (pytest.ini / pyproject.toml / setup.cfg) for test paths and markers, and requirements.txt for whether pytest / pytest-cov are available.
3. Database/model layer access patterns — If the tested code involves a database, confirm: how tables are created (_init_db? migration? _ensure_schema?), how connections are obtained (db.get_connection()? store.db_path? self.conn?).
4. Project incident memory — If the project keeps CLAUDE.md / AGENTS.md / incident docs, scan them for failure patterns that intersect the changed files (a known status-code contract, a route-sync rule, a build quirk). A test guarding a pattern that has actually broken before is worth ten generic ones.
5. Verification inventory — List the project's existing verification assets so the report can route to them by name: smoke/release scripts (scripts/*smoke*, *release*), Makefile test targets, package.json scripts, e2e/monitor test directories. A senior tester knows what verification machinery already exists and never rebuilds it — duplicate coverage wastes the reader's trust.

Environment verification: Pick an existing simple test and run it to confirm the test infrastructure works:

pytest tests/test_something.py::test_simple -v

If even this fails, fix the environment first — don't write new tests.

If test infrastructure cannot be fixed after 2 attempts:

• If pytest not found → Check if it's in requirements.txt; if not, install it (pip install pytest). If install fails, report "Test infrastructure missing and cannot be installed" and exit
• If conftest.py has import errors → Read the error, fix the import. If the missing module is a project dependency, install it. If it's internal code that doesn't exist, report the gap and exit
• If database/schema issues → Check if the test DB path exists and has the right schema. If schema init is broken, report the specific missing table/column and exit
• Never write tests on a broken foundation — a green test on broken infra is worse than no test

Step 3: Analyze Existing Coverage

Find existing tests for the changed files, record covered scenarios.

If the project has no test infrastructure, set it up automatically:

1. Check for pytest in requirements.txt — if missing, add it
2. Create a minimal conftest.py with the project's app factory or client setup
3. Create a tests/ directory if it doesn't exist
4. Run pytest --co to verify the setup works If any step fails, report the specific failure and exit — don't write tests on an unverified foundation.

Step 4: Write Tests

Don't separate "decision" and "execution." After analyzing, write tests directly — don't present an analysis report and wait for confirmation.

For large changes (5+ files), divide and conquer: If your agent supports sub-agent dispatch, use it to parallelize test writing across files. If sub-agents are not available or fail, fall back to sequential processing — test the highest-risk file first, then work down.

Independent auditor (high-risk changes): Authors are the worst reviewers of their own code — their blind spots travel with their reasoning. For high-risk changes (the High row of the risk table: auth, permissions, data isolation, payment, destructive ops), if sub-agent dispatch is available, hand the audit to a fresh subagent passing ONLY the diff, the changed file paths, and any contract doc paths — explicitly not this session's analysis, intentions, or assumed expectations. The subagent forms its own mental model from code and docs, the way an independent human reviewer would; where its conclusions differ from yours, that difference is signal. If dispatch is unavailable, the expected-value provenance rule above is the fallback. When invoked via /change-safety, skip this dispatch — its impact gate (门 2) already runs a de-authored subagent on the same diff, and a second fresh context buys no additional independence, only tokens.

Test code principles:

• Use the project's existing framework and style, reuse conftest helpers — if conftest has make_test_client(), use it; if it has seed_data(), use it; don't create your own fixtures for things that already exist
• Each test verifies one thing, named to describe "what input → what expected"
• Only isolate external dependencies (LLM calls, file I/O, external APIs), don't mock the tested module's internals
• High-risk assertions need provenance: for contract semantics (status-code meanings, cross-service field meanings, payment/auth outcomes), derive expected values from an external source — API docs, design docs, incident write-ups — not from what the implementation currently returns. Cite the source in a test comment (# expected per docs/api-contract.md). Tests written by the same author as the code tend to legalize its bugs; the citation is what breaks the loop.
• React component tests: prefer testing pure logic functions; component tests only verify core interactions; if dependencies are too heavy, skip the component layer and test logic only

After writing tests, self-check: For test data with nested dict/list (especially fake API responses), count the brackets. These syntax errors are the most common and time-wasting low-level mistakes.

Step 5: Run, Debug, Converge

If pytest finds 0 tests after writing → Check: did the test file get saved? Is it in the right directory? Does it match pytest's naming convention (test_*.py)? If the file is correct but pytest still finds 0 tests, the test functions may not be prefixed with test_ — fix naming and retry once.

Strict retry limit: maximum 2 retries per test file.

1st failure → Read error, locate cause, fix
2nd failure → Pause. Don't continue changing, diagnose:
  - Same error as 1st try? → Your fix didn't address the root cause. Back to Step 2: re-read conftest and existing tests
  - Different error? → You fixed one thing but broke another. Check if the test data setup is correct
  - Import/module error? → The test environment is missing dependencies. Check requirements.txt
  - Assertion error on expected values? → The tested code may have a bug. Record it, skip this test
If environment/understanding issue → Re-read conftest + existing tests, understand correct pattern before fixing
If tested code bug → Record bug, skip, continue to next

Never retry blindly. The same error appearing twice means your mental model is wrong — stop and re-understand, don't keep trying.

After discovering a bug, scan for similar issues: If you find a route ordering error, check other routes in the same file for similar issues. If you find a missing column error, check other queries for references to columns missing from the schema. One discovery, batch investigation.

Falsification pass (high-risk changes): When tests are green on a high-risk change (High row of the risk table), re-read the diff once more with the mindset "at least one bug is still hiding here — my job is to find it." For each of the four risk classes — timing/ordering, contract/field semantics, data/edge values, config/environment — ask "how does this break?" once. A verifier mindset confirms what works; a falsifier mindset finds what doesn't — green tests clear low-risk changes but do not clear high-risk ones on their own. Anything the pass turns up becomes a bug entry in the report or one more test, not a silent fix.

Step 6: Report Results

The report answers "can I commit?" for a reader who cannot evaluate test quality themselves. Four layers, in this order — compact, not a technical transcript:

1. Verdict — one line: safe to commit / commit with follow-ups / blocked (bugs found).
2. Coverage map — in plain language, which behaviors are now verified ("payment amount recalculates under promo pricing"). Label each item [behavior] (really executed) or [guard] (source-level assertion only).
3. Not covered here + why — what this test layer cannot reach: cross-service timing, real LLM calls, nginx/env behavior, browser rendering. For each, name the layer that owns verifying it.
4. Next verification step — the single most valuable follow-up, recommended by name from the verification inventory (Step 2), not a generic "run E2E" ("run scripts/release-smoke-test.sh before deploy", "check X on test env"). Unit-green ≠ deploy-ready.

Also state scope: how many files/repos were reviewed, and — in a multi-repo workspace — which repos were audited and which were not.

Assumptions to confirm (standalone runs on business-rule changes): when the change implements a business rule (pricing, gating, permissions, state transitions), append 2-3 assumptions the code makes that you could not back with documentation — phrased for the user to confirm ("assumes the $2 gate uses pre-discount amount", "assumes status 05 means reject, not success"). This is the lightweight version of a senior tester asking "should it even work this way?" Listing, not asking, keeps the zero-interaction mode intact. Skip it when invoked via change-safety — its impact gate runs a stronger version of this question.

If bugs were found → list each bug's location, issue, and suggestion after the four layers.

If coverage was already sufficient with zero gaps → layers 2-4 may collapse to one line ("all changed files covered, no boundary-crossing changes"), but only after Step 1's boundary detection actually came up empty.

Compatibility with Other Skills

When encountering these scenarios, delegate to specialized tools instead of writing yourself:

• Need browser end-to-end testing → Use Playwright or your runtime's built-in browser tool
• Need E2E testing → Use Playwright/Cypress; if your runtime has a built-in E2E skill, use that
• Need security review → Use SAST tools (semgrep, bandit) or your runtime's security review skill

Natural companions (no need to actively call, but complementary):

• Commit workflow — Naturally follows after test-audit passes
• Code review — Review quality first, then test-audit for coverage
• Deploy — test-audit green means segment-level confidence only; the project's real deploy gate is its own smoke/release process (e.g., release-smoke). Point the user there; never let "ready to commit" read as "ready to deploy"
• /change-safety — Calls test-audit as part of its regression gate (门 3). When invoked via change-safety, focus on test coverage and execution; change-safety handles impact analysis and deploy verification. Skip the assumptions list and the independent-auditor dispatch in that mode (change-safety's impact gate owns the stronger version of both — its 门 2 subagent is already the de-authored reviewer for this diff); the verification inventory still runs because the report's next-step layer needs it.

Absolute Don'ts

• Don't ask the user "Which tests to add", "What framework to use", "How deep to cover"
• Don't write snapshot tests — High maintenance cost, low value
• Don't test implementation details — Don't check how many times a function was called, don't check internal state
• Don't write tests that always pass — No assertions, assertions too loose, only checking types not values
• Don't pursue 100% coverage — Critical path coverage is enough
• Don't introduce new test dependencies — Use the project's existing framework
• Don't dump the technical analysis process — the report is verdict + coverage + gaps + next step, not a transcript
• Don't give a bare "ready to commit" — a verdict without the not-covered layer is a false green light
• Don't pause when discovering bugs — Record, continue, report all at the end
• Don't rigidly apply rules — If the code's situation doesn't match the guidelines, use your judgment
• Don't reinvent test helpers — Reuse what's in conftest; only write new ones if none exist
• Don't retry blindly — Must stop and analyze root cause after 2 failures

Common Pitfall Patterns

These are high-frequency failure patterns discovered from real usage. Actively avoid when writing tests:

Feed the Pitfall Table (closing ritual)

If this run hit the same new failure pattern at least twice (a schema/fixture mismatch flavor not in the table, a framework quirk), append one row to Common Pitfall Patterns above and commit it in this skill's own repo (git -C ~/.claude/skills/test-audit commit after adding the row, message like chore: add pitfall — <pattern>). At most one row per run; skip when nothing repeated. This is how the skill learns from real usage.