# Pressure Testing Skills

> Use when validating or hardening a SKILL.md, prompt, rule, or any agent-followed document — when you need proof it actually works, not just that it reads well. Runs a fresh subagent that follows the doc literally on a known-answer fixture, captures friction, and iterates to determinism before minimizing. Use after writing-skills produces a draft, or whenever a skill 'looks right' but hasn't been proven under an agent.

- Skill: `rocm/pressure-testing-skills` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rocm/pressure-testing-skills`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rocm/pressure-testing-skills/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: rocm (https://skillmd.com/u/rocm)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rocm/pressure-testing-skills

---


# Pressure-Testing Skills — amd-smi

Empirically prove a skill (or prompt/rule/checklist) works by making a fresh
subagent follow it **literally** on a fixture whose answers you already know, then
iterate on the failures until the run is deterministic. Reading a skill tells you
it's plausible. Only a literal run by a naive agent tells you it's correct.

**Iron Law: A SKILL IS NOT DONE UNTIL A FRESH SUBAGENT, FOLLOWING IT VERBATIM ON
A KNOWN-ANSWER FIXTURE, CATCHES EVERY PLANTED ISSUE WITH ZERO GUESSING AND NO
BLOCKING FRICTION — WITHOUT INVENTING STEPS THE SKILL DIDN'T GIVE.**

If the agent got the right answer by adding its own check, the skill failed —
that check belongs in the skill. If it "ran clean" but missed a planted issue,
the skill failed. If it had to guess which value to use, the skill failed.

**REQUIRED BACKGROUND:** you must understand `writing-skills` (structure, CSO,
token budget) and `dispatching-parallel-agents` (how to run subagents) first.

## Why Literal + Naive + Known-Answer

The whole method rests on three constraints — drop any one and the test lies:

| Constraint | Drop it and… |
|-----------|--------------|
| **Literal** — agent runs the documented commands verbatim, invents nothing | A clever agent papers over gaps with its own reasoning; you ship a skill only experts can follow |
| **Naive** — fresh subagent, no prior context, no domain memory | Your own knowledge leaks in and the skill looks clearer than it is |
| **Known-answer** — fixture with a pre-written answer key | You can't tell "caught everything" from "got lucky / ran clean" |

## The Loop

```
0. Build the fixture + answer key (the hardest and most important step)
1. Snapshot the skill to a stable temp path  (vN)
2. Dispatch a FRESH, correctly-tooled subagent — follow vN verbatim, invent nothing
3. Collect: issues-caught-vs-answer-key  AND  friction log
4. Fix the ROOT CAUSE of each miss / friction  → snapshot v(N+1)
5. Repeat 2–4 until GREEN (stop condition below)
6. Minimize: cut redundancy, re-test after EACH cut, revert any regression
```

### 0. Fixture + answer key

You cannot grade a test with no answer key. Before iterating, write down every
issue the fixture contains and the exact evidence that should surface it.

- **Best fixture:** a real artifact with known defects (e.g. a `CHANGELOG.md`
  on `develop` you already audited by hand). Real fixtures expose real ambiguity.
- **Answer key:** a list like "F1: entry X is misfiled → commit `abc` merged after
  pin `def`; F2: `### Fixed` is a disallowed heading; F3: deprecation under the
  wrong section." Grade every run against it.
- A skill with no fixture cannot be pressure-tested. If you can't build one, the
  skill is probably too vague to be useful — fix that first.
- **Prefer the LIVE state; historical replays carry time-anchored traps.** If the
  skill resolves any bound from "now" (current tags, `origin/develop`, HEAD), a
  checked-out *past* state gives meaningless results — the bound is anchored to the
  present, not the fixture's era. In one run, auditing a pre-release commit bounded
  a section by a pin that didn't exist yet, and the untagged-release fallback
  (`origin/develop`) pointed at *today's* tip, so the "too new" check could never
  fire and the audit ran clean while proving nothing. Use the live artifact as the
  fixture, or freeze the time-relative inputs too — otherwise you test a mirage.
- **Grade coverage against what the real task touches, not what the skill looks
  at.** If the fixture's defects span the whole file but the skill only inspects
  the top section, a literal run reports "clean" and is structurally blind to most
  of the answer key. A scope mismatch between skill and task is itself a finding.

### 1. Snapshot to a stable path

Copy the skill under test to a temp path (`/tmp/<skill>-test/SKILL-vN.md`) and
point the subagent at ONLY that path. Never point it at the file you're editing
or an installed copy — a stale `.claude/skills/` copy will silently override your
work. Tell the agent explicitly: "ignore any installed copy; use only this path."

### 2. Dispatch a correctly-tooled, fresh subagent

- **Tools must match what the skill's commands need.** A skill full of `git`/`gh`
  shell commands tested by a no-terminal agent proves nothing — it will only catch
  surface issues (headings, wording) and give false confidence on the real logic.
  Use a terminal-capable general subagent for shell-based skills.
- **Fresh context every iteration** so prior runs don't inflate apparent clarity.
- Give it the fixture setup, the snapshot path, and the literal-follow rules.

### 3. Collect two distinct signals

Grade every run on both — they fail independently:

| Signal | Question | A gap here means |
|--------|----------|------------------|
| **Coverage** | Did it catch every answer-key issue, using only documented commands? | The skill is missing a check → add it |
| **Determinism** | Did it have to guess, interpret, or invent anything? | The skill is ambiguous → spell it out |

"Ran clean with no errors" is **not** success. A command can exit 0, leak a
swallowed traceback, and still produce a plausible-but-wrong bound. Ask for the
friction log: every command that failed, was ambiguous, leaked an error, or forced
a guess — quoted with its **exact command and actual output**. The friction log,
not the verdict, is what drives the next iteration.

**Absence of output is not proof.** A clean pass and a silently-broken run look
identical — both print nothing. A check that greps a section and finds no problems,
and a check that ran on *zero* input (wrong path, empty range, swallowed error),
are indistinguishable from the outside. Make the skill emit a positive count
("N items checked, 0 flagged"), and have the test agent confirm the check ran on
real input, not an empty set, before trusting a clean result.

### 4. Fix root causes, not symptoms

Each friction item points at a real defect. Fix the cause:

| Real example from a changelog-skill loop | Root-cause fix |
|------------------------------------------|----------------|
| Audit loop checked only the lower bound → false "all clear" | Check both bounds of the range |
| `<V>`=`7.14.0` built tag `therock-7.14.0` (404) → silent `origin/develop` fallback | Document that tags drop the patch component |
| `2>/dev/null` on `gh` didn't cover the `python3` in the pipe → leaked traceback | Move error handling to cover the whole pipe |
| Agent doubted a correct verdict and nearly overrode it | Add the *why* so a skeptic trusts it (persuasion, not just rules) |
| A clean placement audit and one that ran on zero commits looked identical | Emit a positive count ("N checked, 0 flagged") so a real pass is distinguishable |
| Skill inspected only the top section; the task spanned the whole file | Match the skill's scope to the task, or make the scope an explicit choice |
| Every rule was prose; the agent had to hand-derive a regex for each check | Give an enforced rule a copy-paste command/regex, not just a description |
| A shortcut grep covered 5 of the 7 layers the doc listed, but read as complete | Make the shortcut cover every item the doc enumerates, or name what it skips |
| "When X, do Y" with X ("changes to the public API") undefined → agent over-applied it | Define the trigger precisely; give the boundary cases (function vs enum vs comment) |
| Fixture body was truncated with `...`, so whole body-rule classes couldn't be graded | Give the agent the COMPLETE artifact, never a paraphrase or excerpt |

Silent-failure traps (`|| fallback`, `2>/dev/null`) are the most dangerous:
they turn a wrong answer into a confident one. Make failures loud or document them.

### 5. Stop condition (loop for REAL improvements, optimally)

Stop when a literal run **catches every answer-key issue, with zero guessing, no
blocking friction, and no invented steps.** Not before — "ran clean" is a trap.
Not after — once it's deterministic and complete, more iterations just gold-plate.
Each iteration must fix a concrete failure a subagent actually hit; if you can't
name the failure, stop.

### 6. Minimize last, and guard against regressions

Only after GREEN, cut for size — and **re-test after every cut**, because a cut or
a "cleaner" rewrite can regress:

- Cut redundant tables, checklists, and restated conventions.
- Self-verify cheap mechanical changes locally (run the new regex/command on the
  fixture) before spending a subagent.
- **A proven heuristic beats theoretical completeness.** In one loop a "more
  correct" regex added false positives on the real fixture; the terse original had
  zero. Reverting was the fix. Validate every change against the fixture; revert
  anything that regresses, even if it looks smarter.

## Test-Agent Prompt Template

Reuse this skeleton for every iteration (fill the bracketed parts):

```
Follow this skill LITERALLY and run its commands verbatim. Do NOT use prior
knowledge of [domain]. Do NOT add checks the skill doesn't instruct.

Skill under test (ONLY source of truth): /tmp/<skill>-test/SKILL-vN.md
Ignore any copy at .claude/skills/... — use only the path above.

Setup: [worktree / fixture commands]
Task:  [what to audit/produce using only the skill's documented procedure]

Report back:
1. Judgment calls — did you have to guess or interpret ANYTHING? (KEY QUESTION)
2. Every issue found: exact text, responsible commit/line, which skill step
   flagged it, the fix.
3. Friction log — every command that failed, leaked an error, produced wrong
   output, or was ambiguous. Quote exact command + actual output. Say so
   explicitly if it ran clean with zero guessing. (MOST IMPORTANT)
4. Verbatim key outputs.
5. Exact commands run, trimmed. Then clean up: [worktree remove].
Read-only. Do NOT edit the fixture or the skill.
```

The "KEY QUESTION" and "MOST IMPORTANT" labels matter — without them agents
report a tidy verdict and bury the friction that you actually need.

## Common Mistakes

| Mistake | Why it breaks the test |
|---------|------------------------|
| Testing a shell-heavy skill with a no-terminal agent | Only surface issues surface; core logic untested → false pass |
| No fixture / no answer key | Can't distinguish "caught everything" from "ran clean" |
| Pointing the agent at the file you're editing or an installed copy | Stale copy silently overrides; you test the wrong text |
| Accepting the verdict, skipping the friction log | You miss the ambiguities that make the next reader fail |
| Treating "exit 0 / no errors" as success | Swallowed failures produce confident wrong answers |
| Letting the agent invent its own checks and calling it a pass | The check isn't in the skill — the next agent won't have it |
| Minimizing before it's correct, or not re-testing each cut | You shrink a broken skill, or a cut regresses silently |
| Looping forever / gold-plating after GREEN | Diminishing returns; stop when deterministic and complete |
| Using a historical/checked-out-past fixture for a skill with "now"-anchored bounds | Time-relative inputs resolve to the present → clean-but-meaningless run |
| Trusting empty output as a pass | Clean pass and silently-broken run are indistinguishable without a positive count |
| Grading only what the skill inspects, not what the task touches | Scope mismatch hides most of the answer key behind a "clean" verdict |
| Feeding the agent a truncated or paraphrased fixture | Rules that need the full artifact (body wording, trailing lines) can't be graded |
| Assuming a stated rule is enforceable without a command | Prose-only rules force the agent to invent a check; two agents check differently |
| Trusting a shortcut check that lists fewer items than the doc | It reads as complete while silently skipping layers the doc enumerated |