AI Agents Validation And QA

ai-agents Validation and QA

This repo runs verification-based governance: a claim is true when a gate, test, or command output says it is, not when an agent asserts it. This skill defines the evidence bar for code changes, where tests live and how they are collected, the anti-pattern canon (each entry paid for by a real incident), and the QA gate semantics at session end.

Audience: a zero-context contributor (human or model) about to write, run, or skip tests in this repository.

Triggers

• what counts as evidence
• how do I test this change
• run skill tests
• can I skip QA

Scope Boundaries

Process

Phase 1: Internalize the evidence bar

.agents/governance/TESTING-RIGOR.md is BLOCKING for code changes (its own Status line, TESTING-RIGOR.md:5). The bar, per function you add or change:

Origin story: in PR #1756, the original 20 unit tests gave 24% block coverage; adding negative, edge, and branch tests raised it to 100% and caught real defects the happy-path tests missed (whitespace handling on verdict matching, conditional OTHER-hint emission, type validation). That review created TESTING-RIGOR.md (TESTING-RIGOR.md:3-16). The rule exists because bots (Copilot, CodeRabbit, Gemini) reliably catch what happy-path tests skip, at roughly 10x the cost of writing the tests up front (TESTING-RIGOR.md:81).

Coverage targets by risk tier (AGENTS.md Standards; TESTING-ANTI-PATTERNS.md:112-118): 100% security-critical, 80% business logic, 60-70% docs/glue. "Security-critical" includes secret handling, input validation, command execution, path sanitization, auth checks.

Quality trumps quantity: .agents/governance/TESTING-ANTI-PATTERNS.md bans coverage theater (assertion-free tests), brittle mocks for impossible scenarios, unit-tests-as-only-testing, quantity over quality, and testing entirely after the fact. A test that never failed during development may not be testing anything (TESTING-ANTI-PATTERNS.md:91).

Phase 2: Know where tests live and how they are collected

pytest collects only testpaths = ["tests"] (pyproject.toml [tool.pytest.ini_options].testpaths). Everything else runs explicitly.

Two skill-test locations exist because older skills colocated tests under their shipped directories. New tests belong under tests/skills/NAME/ so customer installs contain runtime assets only. tests/test_skill_bundle_suites_run.py keeps legacy colocated suites running until they migrate.

Useful invocations, all verified (as of 2026-07-03):

uv run pytest tests/ -x                                    # full default suite, stop on first failure
uv run pytest tests/test_ai_review.py -x                   # one file
uv run pytest tests/ -m unit                               # by marker
uv run pytest tests/skills/NAME/ --collect-only -q         # prove new skill tests are collected
uv run pytest .claude/skills/prose-self-check/tests/ -q    # legacy colocated suite

Markers (pyproject.toml [tool.pytest.ini_options].markers): unit, integration, safe_push_transport, security, smoke, windows_path. safe_push_transport means the test touches a non-local transport and is excluded from pre-push. smoke means real-CLI tests needing auth/credits, nightly only; the smoke gate asserts they were not skipped (issue #2231 item 4). windows_path means the test exercises Windows path handling and must run on a Windows runner. Always uv run pytest, never bare pytest or python3 -m pytest outside the venv: PyYAML and friends live in the uv venv (see ai-agents-build-and-env).

Stale doc warning: .agents/governance/test-location-standards.md still describes a Pester/*.Tests.ps1 layout. Zero *.Tests.ps1 files exist (as of 2026-07-03; ADR-042 Python migration). Trust the table above and pyproject.toml, not that file.

Phase 3: Write tests that count

Beyond pos+neg+edge, this repo demands five specific disciplines:

1. Isolation from the real repo. The root conftest.py (repo root, lines 315-386) fails any test that moves the REAL repo HEAD (issue #2316): every git mutation must run in a tmp_path repo with cwd= that repo. Supporting fixtures in tests/conftest.py: GIT_CONFIG_COUNT injection neutralizes host commit.gpgsign so tmp-repo commits work in signing environments (issue #2548, tests/conftest.py:26-61), and AI_AGENTS_PROJECT_REPO=1 defaults identity for guards that check the origin remote (issue #2610, tests/conftest.py:64-76). Consumer-repo simulation tests override that env var to "0".

2. Runtime-contract tests for generated artifacts (FM-11). A generated artifact that has never been executed is not done (FAILURE-MODES.md:28, index row 11; the #2205 incident wedged every plugin customer for 33 days). .claude/rules/generated-artifacts.md:67-73 requires: execute the shipped artifact under the host's real contract (foreign cwd, host-set env vars), assert the intended effect, and include a negative control proving the test CAN fail (a bare relative path must fail the same harness). The exemplar is tests/build_scripts/test_generate_hooks_runtime_contract.py: see test_negative_control_bare_relative_path_fails and test_anchor_is_load_bearing_when_no_plugin_root_var_set.

3. Negative controls beat self-reference. A test that string-matches the generator's own output passes when the generator is consistently wrong. The first #2205 fix shipped exactly this (retro 2026-06-02-pr-2205-customer-wedge-incident.md:49,83) and it hid two more defects. Every contract test needs a case where the wrong artifact fails.

4. Threshold detectors need calibration. Any threshold-based signal (rework count, thread count, file count) must be replayed against roughly the last 5 real merged PRs and shown to fire correctly before shipping. PR #1989's M4 warning shipped with threshold 6 in a repo whose PRs maxed at 4 file edits: it could never fire (retro 2026-05-10-pr-1989-recursive-failure.md:151-157). A detector that cannot fire on recent real work is not calibrated.

5. Guards run on their own branch. A PR shipping a pre-push or pre-commit guard must show that guard running against its own branch, terminal output in the PR description (retro 2026-05-10-pr-1989-recursive-failure.md:131-137; M5 was never applied to the PR that shipped it).

Phase 4: Prove coverage

100% block coverage on changed files, with only defensive-unreachable exclusions (# pragma: no cover) plus written justification (TESTING-RIGOR.md:53).

uv run pytest tests/test_ai_review.py tests/test_verdict.py tests/test_quality_gate.py --cov=scripts.ai_review_common.verdict --cov-branch --cov-fail-under=100

Two traps, both fossilized in .github/workflows/pytest.yml comments:

Related discipline, FM-10: there is no neutral default for a missing signal (FAILURE-MODES.md:387). The neutral default is UNKNOWN, not PASS (extract_verdict on no-match, merge_verdicts on empty; get_verdict fails safe to CRITICAL_FAIL). PR #1965 still took 3 fix rounds to make UNKNOWN blocking everywhere it flowed: the workflow verdict list, the action parser allowlist, and the action exit-code mapping. When testing parsers or gates, always include the missing-signal case and assert it raises or blocks, never that it silently passes.

Phase 5: QA evidence semantics

QA validation remains required for feature code. Session-log presence does not affect that gate. Skip classes are defined by ADR-034 and .agents/architecture/ADR-034-investigation-session-qa-exemption.md:

Explicitly NOT skippable: .agents/planning/, .agents/architecture/ADR-*, .github/, .claude/agents/, src/, and scripts/. Optional session logs, analysis artifacts, and memory updates are filtered when deciding whether QA is required.

Mixed work (investigation turned into code)? Split the commits. Commit investigation work with skip evidence, then validate the code change with real QA.

Record QA evidence in the PR, transcript, per-issue handoff, or an optional session log.

Phase 6: Add tests for a new skill, hook, or script

• Skill: create tests/skills/NAME/test_skill_structure.py for frontmatter, sections, size, and behavior contracts. Do not place new tests inside .claude/skills/NAME/; that tree ships to customers.
• Skill scripts with behavior: add tests/skills/NAME/test_SCRIPT.py; add __init__.py and a conftest.py for sys.path if importing the script by module name.
• Hook: add tests/hooks/test_NAME.py; use tests/hook_test_helpers.py; cover exit 0 (allow, stdout context), exit 2 (block, stdout message), and malformed-stdin input.
• Validation/build script: add tests/test_NAME.py or tests/build_scripts/test_NAME.py; test exit codes per ADR-035 (0 ok, 1 logic, 2 config, 3 external, 4 auth).
• Generated artifact: add a runtime-contract test with a negative control (Phase 3, item 2).
• All git-mutating tests isolated in tmp_path repos (the #2316 guard will fail you otherwise).
• Prove collection: uv run pytest PATH --collect-only -q shows your tests; then run them; then run the coverage pin form from Phase 4 on changed files.
• Run uv run python scripts/validation/pre_pr.py before the PR (the local shift-left aggregate; see ai-agents-change-control for the full gate ladder).

Anti-Patterns

Each row cost real time. Do not re-earn these lessons.

Verification

Before claiming a change meets the evidence bar:

• Every new/changed function has pos, neg, and edge tests; every error and conditional branch exercised
• External I/O mocked in unit tests; CLI exit codes tested where a CLI changed
• uv run pytest tests/ -x green locally, plus explicit runs for any legacy .claude/skills/NAME/tests/ you touched
• Coverage proven at 100% on changed files using the module-name --cov form
• Generated artifacts have a runtime-contract test with a negative control
• Any new threshold or guard shows calibration/self-application evidence in the PR description
• QA row in the session log carries a report path or a legal SKIPPED: evidence string with allowlisted staged files

Provenance and Maintenance

Verified 2026-07-29 against the working tree (issue #3828 re-verification pass; the earlier 2026-07-03 pass had rotted for pyproject.toml, both conftest.py files, .github/workflows/pytest.yml, and .claude/rules/generated-artifacts.md). Sources: .agents/governance/TESTING-RIGOR.md:3-55,77-81, .agents/governance/TESTING-ANTI-PATTERNS.md:9-118, pyproject.toml [tool.pytest.ini_options], repo-root conftest.py:315-386, tests/conftest.py:19-76, .github/workflows/pytest.yml:196-222, .claude/rules/generated-artifacts.md:67-73, .claude/rules/claude-agents.md:18, .agents/architecture/ADR-034-investigation-session-qa-exemption.md:62-110, scripts/validate_session_json.py (_QA_SKIP_CHECKERS), .agents/governance/FAILURE-MODES.md:14-30,387, .agents/retrospective/2026-05-10-pr-1989-recursive-failure.md:110-157, .agents/retrospective/2026-06-02-pr-2205-customer-wedge-incident.md:23-83.

Re-verify volatile facts:

grep -n testpaths pyproject.toml                                  # collection roots
sed -n '315,386p' conftest.py                                     # #2316 HEAD guard still present
grep -n "cov-fail-under" .github/workflows/pytest.yml             # coverage pins and forms
grep -n "_QA_SKIP_CHECKERS" -A5 scripts/validate_session_json.py   # QA skip evidence strings
ls tests/skills/ .claude/skills/prose-self-check/tests/           # both skill-test locations alive
git ls-files '*.Tests.ps1' | wc -l                                # Pester doc still stale if 0

If TESTING-RIGOR.md, ADR-034, or the pytest.yml pins change, update the matching phase here in the same PR.