Audit Tests

audit-tests — Diagnostic Test Auditor (7-Layer + RTM)

Invocation: "audit tests" · "find gaps" · "full sweep" · "7-layer audit" · "rtm check" · "test quality".

Enforcement harness: this skill delegates all deterministic checks (hash-pin, escape-scan, CRAP, architecture, bias, Gherkin lint) to the in-repo installation of @intentsolutions/audit-harness. Node repos install via pnpm add -D @intentsolutions/audit-harness; other languages vendor via the package's install.sh. Never reference ~/.claude/... paths from a target repo's hooks or CI — the enforcement must travel with the code.

Read-only diagnostic skill. Classifies the repo, maps applicable layers from the 7-layer testing taxonomy, runs deterministic gates, builds requirements traceability (RTM / personas / journeys), writes TEST_AUDIT.md + updates tests/TESTING.md, and mandatorily hands off to implement-tests when P0/P1 gaps exist.

No filesystem mutations other than TEST_AUDIT.md (transient) and the observational sections of tests/TESTING.md, tests/RTM.md, tests/PERSONAS.md, tests/JOURNEYS.md.

Overview

Diagnostic-only test suite auditor for any Intent Solutions (or compatible) repo. Classifies the repository, maps the 7-layer testing taxonomy, runs deterministic gates via in-repo @intentsolutions/audit-harness, builds RTM/personas/journeys traceability, and writes TEST_AUDIT.md plus observational sections of tests/TESTING.md. When P0/P1 gaps exist it hands off to implement-tests instead of implementing tests itself.

Prerequisites

• A git working tree for the target repository
• Ability to run Read/Glob/Grep/Bash against that tree
• For deterministic gates: in-repo audit-harness (or install path documented for the language) — never wire target CI/hooks to ~/.claude/...

You are the engineer

Deterministic tools do the grading. The engineer owns the walls (features/*.feature, coverage thresholds, architecture rules, TESTING.md policy sections, RTM.md MoSCoW tags). The AI operates inside them and never grades itself — every gate is a tool that returns exit 0.

Ownership (audit-tests may READ; never modify)

The 7-layer taxonomy (canonical)

Full definitions and gap-fill rationale: {baseDir}/references/taxonomy.md. Classification → applicable layers: {baseDir}/references/layer-applicability.md.

Walls live inside layers. Wall 1 (Acceptance) = L7 spec + L6 glue. Walls 2–7 = L3. See taxonomy.md § "Walls inside layers".

Instructions

Step 0.5 — Harness freshness check (automatic, ≤ 500ms, cache 24h)

Run this early so drift surfaces without remembering to invoke /sync-testing-harness:

CACHE=~/.cache/audit-harness/freshness-check
mkdir -p "$(dirname "$CACHE")"

# Skip if checked in last 24h

if [[ -f "$CACHE" ]] && [[ $(($(date +%s) - $(date -r "$CACHE" +%s))) -lt 86400 ]]; then
  cat "$CACHE"
else
  latest=$(curl -fsS --max-time 3 https://registry.npmjs.org/@intentsolutions/audit-harness/latest 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["version"])' 2>/dev/null || echo "unknown")
  installed=$(pnpm list @intentsolutions/audit-harness --depth=0 --json 2>/dev/null | python3 -c 'import sys,json;d=json.load(sys.stdin);print(d[0].get("devDependencies",{}).get("@intentsolutions/audit-harness",{}).get("version","none"))' 2>/dev/null || echo "none")
  if [[ "$latest" != "unknown" && "$latest" != "$installed" && "$installed" != "none" ]]; then
    msg="⚠ audit-harness drift: installed=$installed → latest=$latest. Run /sync-testing-harness or: pnpm up @intentsolutions/audit-harness"
  else
    msg="✓ audit-harness: $installed (latest: $latest)"
  fi
  echo "$msg" | tee "$CACHE"
fi

• Cache hit (< 24h old) → instant, no network
• Network failure → silently skipped (offline-safe)
• Drift detected → surfaced in TEST_AUDIT.md under ## Freshness
• Not blocking — audit continues regardless

Step 1 — Read context

1. Check for tests/TESTING.md. If absent → proceed to discovery. If present → read ## Classification, ## Thresholds, ## Waived layers. Engineer policy wins.
2. Verify the target repo has @intentsolutions/audit-harness installed (Node: package.json devDep; other langs: .audit-harness/ directory exists). If absent, add to the handoff payload as L0 install. Hooks and CI reference the in-repo harness, never ~/.claude/ paths.
3. Verify hash manifest via pnpm exec audit-harness verify (Node) or scripts/audit-harness verify (vendored). Interpret exit code:
   • 0 (OK) — pinned hashes match; continue.
   • 2 (HARNESS_TAMPERED) — pinned artifact changed; halt the pipeline; flag in TEST_AUDIT.md and do not proceed.
   • 3 (no manifest) — fresh repo or never initialized; continue. Log "no hash manifest — fresh repo" in the report. This is NOT a halt condition.
4. Check current branch. On main / develop / protected branches, any handoff at Step 8 will prompt via AskUserQuestion. On feat/* branches, handoff runs autonomously.

Step 2 — Classify

Deterministic floor first (the harness brain). Run pnpm exec audit-harness classify (Node) / scripts/audit-harness classify (vendored) / audit-harness classify to emit the read-only audit-profile/v1 value — the UNION of detected classifications (service / api / library / cli / frontend / embedded / monorepo + skill / agent / hook / mcp / plugin / marketplace / action), the resolved gate set (a projection of registry.v1.json, recorded by registry_hash), and an explicit unresolved[].

Then dispatch test-discovery-agent (via Task tool) only to refine the unresolved[] residue (ambiguous/novel repos) — it proposes classification patches a human pins in .audit-harness.yml/TESTING.md; it never overrides the deterministic classification value (keeps CI reproducible). Matrix + signals: {baseDir}/references/layer-applicability.md (itself generated from the registry datum via audit-harness gen-layer-applicability).

Monorepo: the profile carries per-package-classify; classify each package independently and run the full pipeline per-package.

Step 3 — Map applicable layers

The applicable gates are already resolved in the audit-profile/v1 from Step 2 (dimension + applicability + enforcement per gate). Dispatch taxonomy-mapper-agent to map presence/absence/configured/enforced on top of the profile's gate set + the TESTING.md compliance overlay.

Output: gap list tagged P0/P1/P2 per layer-applicability.md severity symbols.

Step 4 — Run deterministic quality gates

Run the harness gate-runners — each is read-only and emits gate-result/v1 Evidence Bundle rows the quality-gate-runner-agent consumes:

New gates ship enforcement: advisory; blocking is engineer-pinned in TESTING.md, FP-rate-gated (audit-harness fp-rate, ≤ 5% bar — see docs/gate-promotion.md). Then dispatch quality-gate-runner-agent for the L3 measurement tools and capture exit-code + report:

Gate failure on prod CRAP > 30, test CRAP > 15, project average > 10, architecture violation > 0, or coverage below policy floor → halt the pipeline; record in TEST_AUDIT.md.

Step 5 — Meta-audit (RTM / personas / journeys)

Dispatch three specialists in parallel (Task tool):

• rtm-builder-agent — rebuild tests/RTM.md from Gherkin scenarios, ADRs, product docs, engineer-declared REQs, commit tags. Apply MoSCoW precedence (explicit tag → source default → engineer override → SHOULD fallback). Preserve engineer overrides. See {baseDir}/references/rtm-personas-journeys.md.
• persona-coverage-agent — rebuild tests/PERSONAS.md; compute per-persona flow coverage; flag personas below threshold.
• journey-mapper-agent — rebuild tests/JOURNEYS.md; per-step layer mapping; flag untested steps.

Classify findings:

Step 6 — Escape-scan on any pending diff

Dispatch escape-detection-agent:

bash {baseDir}/scripts/escape-scan.sh --staged    # reads policy floors from tests/TESTING.md

Exit grammar: 0 clean · 1 CHALLENGE · 2 REFUSE. Any REFUSE halts the audit with the specific pattern recorded in TEST_AUDIT.md.

Step 7 — Write reports

1. TEST_AUDIT.md at repo root (transient; short-lived):
   • Grade (letter + numeric)
   • Classification confirmed + applicable layers + waived layers
   • Per-layer presence / config / enforcement table
   • P0/P1/P2 gap list
   • RTM summary (by MoSCoW tier)
   • Persona + journey coverage summary
   • Escape-scan result
2. tests/TESTING.md — update observational sections only (## Installed gates, ## Frameworks, ## Last audit, ## Traceability) via Edit tool, never Write. Policy sections untouched. Schema: {baseDir}/references/testing-md-spec.md.

Step 8 — Mandatory handoff to implement-tests

if any(P0 gaps) OR any(P1 gaps):
    payload = {
        "classification": {...},
        "tests_md_path": "tests/TESTING.md",
        "p0_gaps": [...],
        "p1_gaps": [...],
        "rtm_gaps": [...],         # uncovered MUSTs
        "persona_gaps": [...],
        "journey_gaps": [...],
        "install_order": [L1, L2, L3, ...]   # from scaffold-architect-agent
    }
    if branch in {main, develop, protected}:
        confirm = AskUserQuestion("P0/P1 gaps found. Run implement-tests now?")
        if not confirm: exit with report
    Skill("implement-tests", args=payload)
else:
    print "✓ No P0/P1 gaps. Audit clean."

Handoff pattern mirrors repo-sweep → gist-auditor — no silent escalation, but autonomous on feature branches per global git policy.

Escape-detection response grammar

Patterns that REFUSE (non-exhaustive — full list in {baseDir}/shared-refs/security-testing.md and the script itself):

Deep reference for the full grammar, rationale, and wall derivations: {baseDir}/references/philosophy.md.

Specialist agents (Phase 1 — single-file)

Each lives at ~/.claude/agents/{agent-name}.md. Audit-tests dispatches via Task tool.

Phase 2 (deferred): each agent may be upgraded to a full skill bundle with its own references/ scripts/ evals/ — see the implementation plan's "Phase 2 Roadmap".

Discovery and pre-flight

Full discovery logic (config-file scan, language detection, package-manager detection, monorepo detection): {baseDir}/references/discovery-preflight.md.

Quality gate deep-dives

Catalog of every gate tool across 10 sweep categories (aligned to 7-layer taxonomy): {baseDir}/references/quality-gates-sweep.md.

Per-layer test-quality bias rules, OWASP mapping, mutation interpretation: {baseDir}/references/test-quality-deep-audit.md.

GitHub-scoped audit (CI health, branch protection, coverage-threshold enforcement): {baseDir}/references/github-audit.md.

Output

Clean audit

AUDIT COMPLETE — {repo-name}
Grade:            A (92/100)
Classification:   service
Applicable:       L1, L2, L3, L4-integ, L4-contract, L5-sec, L6-smoke
Waived:           L5-a11y (no UI), L7-UAT
P0 gaps:          0
P1 gaps:          0
RTM:              22 MUST / all covered · 14 SHOULD / 14 covered
Escape-scan:      clean
Report:           TEST_AUDIT.md
TESTING.md:       updated (observational sections)
Handoff:          none needed

Gaps found (feat branch — autonomous)

AUDIT COMPLETE — {repo-name}
Grade:            C (68/100)
P0 gaps:          2 (L3 coverage below 80; REQ-002 MUST uncovered)
P1 gaps:          5
Handoff →         implement-tests (autonomous; feat branch)

Gaps found (main branch — confirmation prompt)

AUDIT COMPLETE — {repo-name}
Grade:            C (68/100)
P0 gaps:          2
P1 gaps:          5
Branch:           main (protected) — prompting for handoff confirmation...

Examples

Example 1 — Fresh Python library, no tests yet. Run audit tests → test-discovery-agent classifies as library, no TESTING.md present → taxonomy-mapper-agent reports L1, L2, L3 all absent (P0), L4–L7 waived per library applicability → rtm-scaffolder schedules skeleton creation → handoff payload: install_order: [L1, L2, L3] → autonomous handoff fires (feat branch) → implement-tests scaffolds pytest + coverage + mutmut + ruff + pre-commit + starter tests.

Example 2 — Mature service, previously audited. Run full sweep. All layers report installed; coverage 82% (above 80 floor); mutation kill rate 74% (above 70 floor); CRAP max 22 (below 30). Grade: A (92/100). TESTING.md#Last audit updated with today's date. No handoff.

Example 3 — Monorepo with three packages. Run audit tests. test-discovery-agent returns monorepo: true, packages: [a, b, c]. Each package classified independently (packages/a = service, packages/b = library, packages/c = cli). Each gets a per-package TESTING.md. Aggregate TEST_AUDIT.md at repo root has three sections. Handoff payload includes per-package install_order arrays.

Example 4 — PR lowers coverage threshold to bypass gate. A PR proposes [tool.coverage.report] fail_under = 60 in pyproject.toml. TESTING.md#Thresholds has coverage.line: 80. escape-detection-agent runs escape-scan.sh --staged. The script reads floor 80 from TESTING.md, sees fail_under dropped to 60, emits [REFUSE] coverage fail_under lowered below policy floor (80), exits 2. Audit halts. No TEST_AUDIT.md produced. No handoff. Engineer must revert the threshold edit or run harness-hash.sh --init to legitimize a policy change.

Example 5 — HIPAA-compliant service with uncovered SHOULD. Repo has Compliance overlay: HIPAA in TESTING.md. RTM.md has one uncovered SHOULD (error-message retry logic). Normally P1 advisory, but the regulated overlay promotes it to P0. Handoff fires with p0_gaps: [{layer: L3, gap: REQ-034 retry logic uncovered (SHOULD, HIPAA overlay)}]. On main branch → AskUserQuestion prompts; engineer approves; implement-tests proceeds.

Troubleshooting

Edge cases

• Missing framework: record gap; skip the measurement; emit install_order: [framework] in handoff.
• Docker unavailable for integration tests: skip L4 integration measurement; still report the gap as "unmeasured — Docker missing".
• Hash mismatch: HARNESS_TAMPERED — halt before any other work; do not write TEST_AUDIT.md.
• Engineer-declared waiver: any layer listed in TESTING.md#Waived layers is skipped silently; appears in report under "Waived (per engineer policy)".
• Regulated overlay: HIPAA/SOX/PCI/SOC2/GDPR/FedRAMP tag in TESTING.md promotes SHOULD uncovered to P0 — see rtm-personas-journeys.md § "Regulated overlay".

Resources

• {baseDir}/references/taxonomy.md — 7-layer canonical definitions
• {baseDir}/references/layer-applicability.md — classification → layer matrix
• {baseDir}/references/testing-md-spec.md — tests/TESTING.md schema
• {baseDir}/references/rtm-personas-journeys.md — RTM / personas / journeys spec
• {baseDir}/references/discovery-preflight.md — discovery and pre-flight checks
• {baseDir}/references/github-audit.md — repo-scoped audit engine
• {baseDir}/references/test-quality-deep-audit.md — bias, mutation, OWASP
• {baseDir}/references/quality-gates-sweep.md — tool catalog aligned to taxonomy
• {baseDir}/references/philosophy.md — Seven Walls derivations, escape-grammar state machine, Human/AI ownership rationale
• {baseDir}/scripts/arch-check.sh — architecture rule dispatcher
• {baseDir}/scripts/bias-count.sh — bias-pattern counter
• {baseDir}/scripts/crap-score.py — CRAP calculator
• {baseDir}/scripts/gherkin-lint.sh — advisory Gherkin quality check
• {baseDir}/scripts/escape-scan.sh — diff-based escape-attempt detector (reads floors from tests/TESTING.md)
• {baseDir}/shared-refs/harness-hash.sh — SHA-256 manifest verifier
• {baseDir}/shared-refs/*.md — per-concern deep references (frameworks, architecture, security, etc.) shared with implement-tests