Pressure Testing Skills

Pressure-Testing Skills — amd-smi

Empirically prove a skill (or prompt/rule/checklist) works by making a fresh subagent follow it literally on a fixture whose answers you already know, then iterate on the failures until the run is deterministic. Reading a skill tells you it's plausible. Only a literal run by a naive agent tells you it's correct.

Iron Law: A SKILL IS NOT DONE UNTIL A FRESH SUBAGENT, FOLLOWING IT VERBATIM ON A KNOWN-ANSWER FIXTURE, CATCHES EVERY PLANTED ISSUE WITH ZERO GUESSING AND NO BLOCKING FRICTION — WITHOUT INVENTING STEPS THE SKILL DIDN'T GIVE.

If the agent got the right answer by adding its own check, the skill failed — that check belongs in the skill. If it "ran clean" but missed a planted issue, the skill failed. If it had to guess which value to use, the skill failed.

REQUIRED BACKGROUND: you must understand writing-skills (structure, CSO, token budget) and dispatching-parallel-agents (how to run subagents) first.

Why Literal + Naive + Known-Answer

The whole method rests on three constraints — drop any one and the test lies:

The Loop

0. Build the fixture + answer key (the hardest and most important step)
1. Snapshot the skill to a stable temp path  (vN)
2. Dispatch a FRESH, correctly-tooled subagent — follow vN verbatim, invent nothing
3. Collect: issues-caught-vs-answer-key  AND  friction log
4. Fix the ROOT CAUSE of each miss / friction  → snapshot v(N+1)
5. Repeat 2–4 until GREEN (stop condition below)
6. Minimize: cut redundancy, re-test after EACH cut, revert any regression

0. Fixture + answer key

You cannot grade a test with no answer key. Before iterating, write down every issue the fixture contains and the exact evidence that should surface it.

• Best fixture: a real artifact with known defects (e.g. a CHANGELOG.md on develop you already audited by hand). Real fixtures expose real ambiguity.
• Answer key: a list like "F1: entry X is misfiled → commit abc merged after pin def; F2: ### Fixed is a disallowed heading; F3: deprecation under the wrong section." Grade every run against it.
• A skill with no fixture cannot be pressure-tested. If you can't build one, the skill is probably too vague to be useful — fix that first.
• Prefer the LIVE state; historical replays carry time-anchored traps. If the skill resolves any bound from "now" (current tags, origin/develop, HEAD), a checked-out past state gives meaningless results — the bound is anchored to the present, not the fixture's era. In one run, auditing a pre-release commit bounded a section by a pin that didn't exist yet, and the untagged-release fallback (origin/develop) pointed at today's tip, so the "too new" check could never fire and the audit ran clean while proving nothing. Use the live artifact as the fixture, or freeze the time-relative inputs too — otherwise you test a mirage.
• Grade coverage against what the real task touches, not what the skill looks at. If the fixture's defects span the whole file but the skill only inspects the top section, a literal run reports "clean" and is structurally blind to most of the answer key. A scope mismatch between skill and task is itself a finding.

1. Snapshot to a stable path

Copy the skill under test to a temp path (/tmp/<skill>-test/SKILL-vN.md) and point the subagent at ONLY that path. Never point it at the file you're editing or an installed copy — a stale .claude/skills/ copy will silently override your work. Tell the agent explicitly: "ignore any installed copy; use only this path."

2. Dispatch a correctly-tooled, fresh subagent

• Tools must match what the skill's commands need. A skill full of git/gh shell commands tested by a no-terminal agent proves nothing — it will only catch surface issues (headings, wording) and give false confidence on the real logic. Use a terminal-capable general subagent for shell-based skills.
• Fresh context every iteration so prior runs don't inflate apparent clarity.
• Give it the fixture setup, the snapshot path, and the literal-follow rules.

3. Collect two distinct signals

Grade every run on both — they fail independently:

"Ran clean with no errors" is not success. A command can exit 0, leak a swallowed traceback, and still produce a plausible-but-wrong bound. Ask for the friction log: every command that failed, was ambiguous, leaked an error, or forced a guess — quoted with its exact command and actual output. The friction log, not the verdict, is what drives the next iteration.

Absence of output is not proof. A clean pass and a silently-broken run look identical — both print nothing. A check that greps a section and finds no problems, and a check that ran on zero input (wrong path, empty range, swallowed error), are indistinguishable from the outside. Make the skill emit a positive count ("N items checked, 0 flagged"), and have the test agent confirm the check ran on real input, not an empty set, before trusting a clean result.

4. Fix root causes, not symptoms

Each friction item points at a real defect. Fix the cause:

Silent-failure traps (|| fallback, 2>/dev/null) are the most dangerous: they turn a wrong answer into a confident one. Make failures loud or document them.

5. Stop condition (loop for REAL improvements, optimally)

Stop when a literal run catches every answer-key issue, with zero guessing, no blocking friction, and no invented steps. Not before — "ran clean" is a trap. Not after — once it's deterministic and complete, more iterations just gold-plate. Each iteration must fix a concrete failure a subagent actually hit; if you can't name the failure, stop.

6. Minimize last, and guard against regressions

Only after GREEN, cut for size — and re-test after every cut, because a cut or a "cleaner" rewrite can regress:

• Cut redundant tables, checklists, and restated conventions.
• Self-verify cheap mechanical changes locally (run the new regex/command on the fixture) before spending a subagent.
• A proven heuristic beats theoretical completeness. In one loop a "more correct" regex added false positives on the real fixture; the terse original had zero. Reverting was the fix. Validate every change against the fixture; revert anything that regresses, even if it looks smarter.

Test-Agent Prompt Template

Reuse this skeleton for every iteration (fill the bracketed parts):

Follow this skill LITERALLY and run its commands verbatim. Do NOT use prior
knowledge of [domain]. Do NOT add checks the skill doesn't instruct.

Skill under test (ONLY source of truth): /tmp/<skill>-test/SKILL-vN.md
Ignore any copy at .claude/skills/... — use only the path above.

Setup: [worktree / fixture commands]
Task:  [what to audit/produce using only the skill's documented procedure]

Report back:
1. Judgment calls — did you have to guess or interpret ANYTHING? (KEY QUESTION)
2. Every issue found: exact text, responsible commit/line, which skill step
   flagged it, the fix.
3. Friction log — every command that failed, leaked an error, produced wrong
   output, or was ambiguous. Quote exact command + actual output. Say so
   explicitly if it ran clean with zero guessing. (MOST IMPORTANT)
4. Verbatim key outputs.
5. Exact commands run, trimmed. Then clean up: [worktree remove].
Read-only. Do NOT edit the fixture or the skill.

The "KEY QUESTION" and "MOST IMPORTANT" labels matter — without them agents report a tidy verdict and bury the friction that you actually need.

Common Mistakes