Audit

Purpose

A test that cannot fail is a false coverage claim: it reports green whatever the code does. This skill runs a deterministic script detector over the suite. It does not execute tests or judge membership, but reports every test the rules v1 catch, with a coverage denominator so "no findings" is never confused with "scanned nothing".

Boundaries, each an incumbent this skill deliberately does not duplicate:

• mutation-testing:audit proves dynamically that tests fail to detect change. It executes mutants, costs real runtime, and judges survivors. This skill is the static complement: cheap AST-level detection of tests that cannot fail by construction. Complements, not rivals.
• check-discriminating-test-skips.sh (this marketplace repo's own CI gate) owns the fourth can't-fail shape, a skip vacating the only discriminating assertion of a case group, for bash *.test.sh. That rule is deliberately absent here; bash test files are out of scope v1.
• The repair queue is out of scope: findings propose an assertion (repair, not pruning: deleting a useless test removes the false claim and the coverage together); applying repairs belongs to the remediation lanes.

Rules v1

Rule ids are the qualified detector-findings form; thresholds are fixed per rule, and every finding states the fired condition in the run's own values.

• Tier is looked up from each rule's row in the detector-findings severity crosswalk (the contract cited under Persisting findings). IMPORTANT on every row, flat per producer. The argument for each mapping lives in the crosswalk row, not here; a per-finding tier choice is exactly what the rule-keyed lookup forbids.
• mock-only-oracle omits Confidence and is advisory by default. The pattern match is certain; its defect-hood is not. Deliberate interaction-style (London-school) tests are the known benign case. Per the detector-findings contract the field is high or omitted, never low.
• Detection bias: every heuristic errs toward not firing. Assertion tokens match generously (a helper named assertValidSum or checkInvariant counts), strings/comments are masked first, skipped tests are not judged. A missed defect costs one finding; a false positive costs the detector its audience.

Running the detector

bash "${CLAUDE_PLUGIN_ROOT}/skills/audit/scripts/cant-fail-scan.sh"            # report + denominator
bash "${CLAUDE_PLUGIN_ROOT}/skills/audit/scripts/cant-fail-scan.sh" --check    # gate: exit 1 findings, 2 gap, 0 clean
bash "${CLAUDE_PLUGIN_ROOT}/skills/audit/scripts/cant-fail-scan.sh" --findings # findings file on stdout

Scan root: the current repo's git toplevel (or $CANT_FAIL_SCAN_ROOT to narrow/point explicitly, a supported operator lever). Ecosystems v1: JS/TS (*.test.*/*.spec.*), Python (test_*.py/*_test.py), C# (*Test.cs/*Tests.cs).

Present the script's findings and its coverage block as reported, the denominator is what makes a clean report a claim rather than an absence. A run that examined 0 test files says so and is never presented as a clean bill.

Gate mode (--check). Fail closed

The machine-checkable gate the liveness-assertion contract "Core contract" fail-loud limb requires of an advisory surface's gating form:

• exit 1, a gating rule fired (zero-assertion, recomputed-expectation; --strict adds mock-only-oracle).
• exit 2, the scan could not run, could not fully read its inputs (unresolved root, unreadable test files, walk errors), or examined 0 test files, a wrong or empty scan root and a healthy suite must not share an exit code. An unread input is never a clean one.
• exit 0. Only a fully read, finding-free scan of at least one test file.

Persisting findings (--persist-findings)

Bare invocation reports and stops, the audit verb's read-only contract. Under the explicit --persist-findings override, also write the findings file the review:fanout fix action consumes:

1. Read the producer contract before the first write. https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/docs/conventions/detector-findings/README.md. It owns the shape's authority, where the file goes, the producer-computed fields, the coexistence obligations, the self-ignore guard, and what a minimal producer may omit. Where the two disagree, the contract wins and this file is the defect. If it cannot be fetched, do not write. Report and stop; a guessed destination reports success while the consumer never scans that path.
2. Resolve the destination per the contract "Where the file goes": run the whole rung order, take the non-interactive collapse where this context cannot ask or persist config, and honor the self-ignore guard including its invalid-root rule.
3. Generate the content with cant-fail-scan.sh --findings (it computes branch: verbatim from git, date: at write time, per-rule Tier/Confidence, repo-relative Location, cell escaping, and the ## Surfaces coverage line; it omits tier:, ## By dimension, and ## Unparsed. No analogue, omit rather than fabricate). Write it to <resolved-dir>/${TS}-cant-fail-tests.md with TS="$(date -u +%Y%m%dT%H%M%SZ)".
4. Never overwrite an existing path. Take -2, -3… (smallest free integer >= 2).
5. A run that examined test files and found nothing still persists: the empty table plus ## Surfaces is coverage the consumer merges. A run that examined nothing writes nothing. The script already refuses --findings there.
6. Re-runs write what they currently find; never replay a previous file's rows.

Exemptions

A deliberate case is recorded in-file: cant-fail-ok: <reason> on the test's declaration line, the line above it, or inside the body, the same recorded-decision shape as the repo gate's discriminating-skip-ok. Exemptions are counted in the coverage block, never silent.

What this skill does NOT do

• Edit, repair, or delete tests. Findings propose an assertion; the repair itself is the remediation lanes' work (/testing:write for authoring, the review fix pass for applying).
• Execute the suite. /toolchain:check runs tests; mutation-testing:audit executes mutants.
• Audit bash *.test.sh, the discriminating-skip repo gate owns that shape.
• Write anything on bare invocation. Persisting is only ever behind --persist-findings.

Next

• A finding names a test that needs a real assertion: /testing:write.
• Findings are persisted with --persist-findings: /review:fanout fix.

Gotchas

• Interaction-style tests trip mock-only-oracle by design. That is why it is advisory in --check and carries no Confidence. A team that asserts interactions deliberately annotates cant-fail-ok: or leaves --strict off; a team that considers them defects gates with --strict.
• Generous assertion tokens buy false-negative risk: a test whose only "assertion" is a helper named checkout() is suppressed by the check token. That is the chosen direction; do not tighten the token list to chase recall at precision's expense.
• recomputed-expectation v1 is the decidable core. Textually identical actual/expected on one line (chains spanning lines are deliberately not matched, and only the first expect per line is examined). x = f(a); assert x == f(a) and C#'s generic Assert.Equal<T>(a, a) are the same defect and are not detected.
• The JS regex-literal masker triggers only after an operator or opening delimiter, never after an identifier, so a regex directly after return is not masked. Wrongly reading division as a regex would mask real code, which is the worse direction. The known cost of that narrow set is a premature-block-closure false positive when an unmasked regex after return contains a brace (e.g. return /}/; inside a test). brace_delta treats the } as code and closes the test before later assertions, so rule-zero-assertion can fire on a body that still has assertions.
• Skips are honored at both levels. Test-level (it.skip/x-prefixed/@skip/[Fact(Skip=…)]) and suite-level (xdescribe/describe.skip/context.skip/suite.skip): a test that does not run is not judged.
• Fixture corpora under evals/fixtures/ are pruned, a detector's planted-defect fixtures are not the consumer's defects. Point $CANT_FAIL_SCAN_ROOT at one explicitly to scan it.
• Platform-conditional skips are outside the detector's reach. A visible skip is not an assertion-free body, so a case that only another platform executes is neither a finding nor coverage here; a green local run is not evidence about it.