dtsong mcs-validation-and-qa
Overview
Evidence here is a passing command with a captured exit code, or a flipped and cited acceptance-contract row, never a visual check. Two things are true at once: strong contract and eval machinery exists, and its live enforcement surface is narrower than the specs imply (no CI governance, 0 required reviews). Both facts must inform every claim of "done."
When to use, and when NOT to
Use this to decide what evidence a change needs, to add a test/eval/AC, or to interpret a pre-commit/CI/gate result correctly.
Route instead to:
- mcs-diagnostics-and-tooling: how to run and interpret a specific diagnostic script (this skill only says which command counts as evidence).
- mcs-failure-archaeology: why an enforcement gap exists, what was tried and reverted (the registry-telemetry saga in full).
- mcs-change-control: commit/PR conventions and non-negotiables (assumed here, not re-taught).
- mcs-architecture-contract: why the acceptance-gate hook is shaped PreToolUse/exit-2 (design rationale, not usage).
The verification surface, honestly stated (2026-07-05)
| Layer | Scope | Notes |
|---|---|---|
| pre-commit hooks | skill-governance scoped to ^skills/.*\.md$; two more hard hooks sit outside it |
Skill-governance: 4 hard (check_frontmatter, check_references, check_isolation, check_context_load) + 1 hard commit-msg stage + 2 warn (check_token_budget, check_prose, always exit 0). As of 2026-07-05 (82292e1, US-005/US-006): model-routing (hard, ^skills/council/model-routing\.json$) and settings-schema (hard, ^settings\.json$) also exist. .claude/ is still outside every glob: never governance-checked. |
CI (validate.yml) |
repo-wide | bash -n on install.sh/hooks/*.sh/scripts/*.sh, python3 -m json.tool on settings.json/hooks.json, installer smoke test, gitleaks. Zero governance hooks, zero pytest. |
| Branch protection (main) | pull requests | Required checks: validation (ubuntu-latest), validation (macos-latest), secret-scan. 0 required approving reviews. pre-commit run --all-files is never re-run by CI; git commit --no-verify bypasses it entirely. |
| pytest | mcp/openrouter/ + session test-stubs/ |
32 tests in mcp/openrouter/tests/, plus test_acceptance.py per council session. Real, runnable, not wired into CI. |
| Type-check | none | Bash/python/markdown repo, no tsc/eslint surface. State that explicitly (owner CLAUDE.md directive 2) rather than run or claim one. |
Run the actual local gate:
pre-commit run --all-files
python3 -m pytest mcp/openrouter/tests/ -q # 31 passed, 1 skipped (live call, network-gated)
pre-commit run --all-files only sees git-tracked files. skills/soc-security,
skills/resume-tailor, skills/ece are gitignored local symlinks (this
machine only); running a hook directly against them still exits 1, but that
can never block a commit here since the path is never staged.
Acceptance-contract discipline
Format (commands/_council-engine.md:1038, live example: the active council
session's acceptance-contract.md): a Quality Gates table, per-AC blocks
(Method, Test, Status, Evidence, Verified-by), then a
## Verification Summary table the gate hook actually parses, ending
Progress: N/M verified. Status is exactly one of pending, verified,
failed, pending-manual. BDD stubs generate per AC into test-stubs/,
each raising Not implemented, AC-NNN pending so the suite starts RED.
How to verify an AC properly:
- Run the exact command in the AC's
Test:field. - Only on a real pass, flip
Statusin both the detail block and the summary table (the gate hook reads only the summary; a mismatched detail block is a lie left in the file). - Fill
Evidencewith what ran and its result, not a description of intent. FillVerified-bywith the issue/PR that did it. - Never hand-edit a row to
verifiedwithout a run. The discipline is entirely on you; nothing checks whether the row is true. - Adding a new AC to a
lockedcontract is a scope change to a negotiated agreement: treat it as needing owner sign-off first (inferred, owner-unconfirmed, no script enforces this).
Gate mechanics (hooks/acceptance-gate.sh, PreToolUse on TaskUpdate):
fires only when tool_input.status == "completed", selects the
most-recently-modified contract across the known session/PRD locations,
skips one untouched for ACCEPTANCE_GATE_STALE_HOURS (default 72h), counts
| pending | plus | failed | rows, and if that sum is nonzero, denies with
exit 2 and a stderr list of unverified criteria. pending-manual never
blocks. It gates only TaskUpdate -> completed, not git commit or gh pr merge.
Eval tiers and trigger evals
| Tier | Method | Cost | When |
|---|---|---|---|
| 1 | pre-commit hooks (static) | free | every commit |
| 2 | claude -p execution of eval cases against Must-Pass criteria |
paid API | before shipping |
| 3 | LLM-as-judge: Clarity/Completeness/Actionability, 1-5 each | paid API | before shipping |
Trigger-eval minimum for a skill's eval-cases/trigger-evals.json: 5
positive cases (should activate) plus 3 negative cases (should not, with
expected_skill: null and the correct routing target noted). As of
2026-07-02, exactly 9 council departments carry trigger-evals.json:
alchemist, architect, cipher, forge, guardian, oracle, prover, skeptic, warden.
pipeline/scripts/run-evals.sh --tier 2 --targets <skill> --model sonnet # or --tier all / --diff
pipeline/scripts/judge-skill-quality.sh --targets <skill> # --diff for changed-only
Both shell out to the claude CLI, cost real API spend per run, and write
results to eval-results/tier2-<timestamp>/results.jsonl or
eval-results/judge-<timestamp>/results.jsonl. Read the JSONL; do not judge
by terminal scroll. Per governance spec 7.4, budget disputes are settled
empirically: run evals at current length, refactor to target, run again, and
keep the shorter version only if the pass rate holds. A shorter skill that
fails more eval cases is a regression, not a win.
How to add evidence
Full recipes, conventions, and gotchas: references/adding-evidence.md.
Summary: a pytest to mcp/openrouter/tests/ imports the module directly
(conftest.py puts mcp/openrouter/ on sys.path), stays network-free by
default, gating any live call behind
@pytest.mark.skipif(os.environ.get("X") != "1"). An eval case is 5-7 cases
per skill with Must Pass/Should Pass/Bonus tiers, plus the trigger-eval
minimums above. A new pre-commit validator goes in pipeline/hooks/, wires
into .pre-commit-config.yaml with the right files: glob and tier, and
must land in .github/workflows/validate.yml too, given 0 required reviews
and universal --no-verify bypass; none of the 6 skill-governance hooks do
this today (owner-unconfirmed whether that gap is accepted).
The golden and certified inventory
Full table (component, evidence type, re-run command):
references/inventory.md. Headline numbers, verified 2026-07-06:
mcp/openrouter/tests/ = 32 tests (31 pass, 1 skipped without
OPENROUTER_SMOKE=1; test_check_models.py added by PR #70); the
config-optimization session's test_acceptance.py = 12 passed, zero
remaining stubs; that session's acceptance contract closed at 26/26
verified (2026-07-06).
Acceptance-threshold discipline
Success is always a number or an exit code: pytest -q status, a grep -c
count, a jq equality check, the contract's Progress: N/M line. Never "ran
it and it looked right." The cautionary example lives in this repo:
skills/council/registry.json usage counters were nearly used to justify
pruning council skills on 0/67 committed uses, because every nonzero count
was uncommitted session-local noise that git checkout . would have erased.
The instrument was correctly distrusted only because someone ran git show HEAD:<path> instead of cat <path>. Treat any counter or telemetry file as
untrustworthy until its number survives that check.
Gotchas
- A clean
pre-commit run --all-filessays nothing aboutskills/soc-security/,skills/resume-tailor/,skills/ece/: gitignored symlinks excluded from every check, while still failingcheck_context_load.py/check_isolation.pyrun directly. - Suite ceiling is contested in docs, not enforcement:
pipeline/config/budgets.jsonand_utils.pyuse 5,500 tokens; rootCLAUDE.md:57says 5,000, a stale doc, not a second live rule. check_token_budget.pyandcheck_prose.pyalways exit 0. "Pre-commit passed" is not "no budget or prose issues": read the WARNING/INFO lines.- CI checks JSON and shell syntax only. A PR can merge with every governance
hook broken and zero pytest run, as long as
settings.json/hooks.jsonparse and the installer smoke test passes. - The acceptance gate fires only on
TaskUpdate -> completed. Marking work done in a commit message or in prose is not gated at all. - A
Verified-byfield citing a closed issue/PR is only real evidence if that issue/PR's own test run is inspectable; trace it back once instead of trusting the URL. is_excluded()inpipeline/hooks/_utils.pyskips path segments namedpipeline,eval-cases,node_modules,.github,templates: any markdown files inside a skill's own templates directory are never checked by any hook.
Provenance and maintenance
Last verified 2026-07-05 by direct command execution against commit
82292e1 plus the then-in-flight working tree (that campaign completed
2026-07-06; main is now at or past 74e34c5).
Re-verify:
- Pre-commit scope (all hooks, not just skill-governance):
grep -B2 "files:" .pre-commit-config.yaml - CI scope:
cat .github/workflows/validate.yml - Branch protection:
gh api repos/dtsong/my-claude-setup/branches/main/protection - openrouter pytest count:
pytest mcp/openrouter/tests/ -q - Gate mechanics:
cat hooks/acceptance-gate.sh - Live contract progress:
grep -n "^Progress:" .claude/council/sessions/*/acceptance-contract.md - Eval-case department count:
find skills/council -name trigger-evals.json | wc -l - Suite ceiling constant:
grep max_simultaneous_tokens pipeline/config/budgets.json - soc-security/resume-tailor symlink status (this machine only):
git check-ignore -v skills/soc-security skills/resume-tailor - Governance spec section numbers:
grep -n "^## 7\.\|^## 8\." pipeline/specs/SKILL-GOVERNANCE-SPEC.md