Exercise Asb Skill

exercise-asb-skill — wind-tunnel for asb-* skills

A skill that reads well can still wield badly: it recites the framework instead of applying it, nods along with weak answers, forgets to refuse the user it should refuse, or produces advice any consultant could have given. The only way to find those failures is to watch the skill being used. This skill simulates that use and judges the result.

Two contexts, same procedure:

• Forging (inside create-asb-skill, Phase 7): exercise the on-disk draft, patch FAILs directly with Edit, re-run failed scenarios.
• Iterating (standalone): exercise a published skill before and/or after proposed edits. Propose patches as diffs and get Jason's sign-off before editing a published skill.

Step 1 — Resolve the target and quarantine

Resolve the target SKILL.md:

• A skill name (asb-rude-qa) → .claude/skills/<name>/SKILL.md.
• An explicit path → that file.
• Neither given → ask.

Read the file ONCE, fully. Then adopt the quarantine: for the rest of the exercise, the wielder side of every role-play may use ONLY what is in that file. No knowledge of Jason's corpus, no memory of the forging conversation that produced the skill, no familiarity with this repo. If the wielder needs a definition, rule, or example that isn't in the file, that absence is itself a finding — note it; do not fill it in from outside knowledge. State the quarantine in one line before the first transcript.

(Recommended default, proven across the asb-interview-* family: dispatch each scenario to a subagent whose prompt names ONLY the SKILL.md path and the scenario — quarantine enforced structurally rather than by discipline, and scenarios run in parallel. Give each subagent explicit PASS/FAIL test questions for the scenario's purpose, and require its final message to be exactly: the transcript, then an "AMBIGUITIES/GAPS NOTED" list of places the wielder needed something the file didn't provide. Those ambiguity lists are where most patches come from — often more than the verdicts. This skill is dev-only; subagents are allowed.)

Step 2 — Design the scenarios

Construct 3–5 scenarios tailored to the target skill's domain. Required mix:

1. Canonical user. Rich, realistic input squarely inside the skill's sweet spot. Tests the happy path end to end.
2. Terse/underspecified user. A two-line prompt with most context missing. Tests whether the skill's intake questions and dwell rules fire — or whether the wielder fills in the blanks itself and barrels ahead.
3. Boundary case. A situation near the edge of where the framework applies. Tests whether the skill handles the edge honestly or confidently misapplies.
4. Refusal case. A user the skill should refuse per its own refusal conditions (vibe-only target, validation-seeker, theater). Tests that refusal actually fires, and fires kindly.
5. (Optional) Lazy answerer. A cooperative user whose answers are consistently weak — "should be fine," "we'll figure it out later." Tests that the bar doesn't drop across many rounds.
6. Die-and-resume (required when the skill maintains a working file). A two-part scenario: the session ends abruptly mid-process; a NEW session (wielder has no memory — only the SKILL.md and what it reads from disk) is asked to finish. Tests that the file alone carries the state: correct pickup point, no re-eliciting of settled items, in-progress note removed at finalization.

Also re-run the exercise after any substantial redesign of an already-exercised skill — earlier passes certify the old text, not the new one. And when Jason reports a live-use failure, treat it as a missing rubric criterion or scenario here, not just a patch to the one skill: the wind tunnel only catches what it tests for.

Each scenario gets a one-line persona (who they are, what they bring, how they behave under pressure). Show the scenario list to Jason before running; he may swap or sharpen one. Don't wait long — if he's clearly in "just run it" mode, proceed.

Step 3 — Run the transcripts

For each scenario, role-play BOTH sides for roughly 6–12 turns, written out as a labeled transcript:

USER: <what they say>
WIELDER: <what the skill-loaded LLM says, per the SKILL.md and nothing else>

Rules of play:

• The simulated user is realistic, not a softball: weak answers sometimes, pushback, mild frustration, occasionally trying to skip steps or get the deliverable without the work.
• The wielder follows the SKILL.md literally — including its phase order, artifact instructions, and refusal conditions. Where the SKILL.md is ambiguous, play the most plausible reading and flag the ambiguity.
• Run the transcript until the scenario's question is answered (the refusal fired or didn't; the dwell held or collapsed) — full exercises don't need to reach their natural end once the verdict is clear.
• Keep transcripts honest. The exercise is worthless if the wielder is played as an idealized version of what the SKILL.md merely gestures at.

Step 4 — Judge each transcript

Judge against this fixed rubric — each criterion is a separate lens with its own verdict line: PASS / WEAK / FAIL plus a quoted line from the transcript as evidence. Skip criteria a scenario can't speak to (mark n/a).

Step 5 — Report and patch

1. Verdict table — scenarios × criteria, one glance.
2. Patch list — every FAIL and WEAK mapped to the SKILL.md section that caused it, with a concrete proposed edit. Distinguish the two failure sources: the SKILL.md is missing/ambiguous (patch the file) vs. the SKILL.md is clear but easy to drift from (strengthen the standing rule — usually by making it more explicit, earlier, or repeated at the point of use).
3. Apply patches:
   • Forging context: Edit the draft directly, then re-run ONLY the failed scenarios until clean.
   • Published skill: show Jason the proposed diffs and get sign-off before editing. After editing, re-run failed scenarios. Remind him bun run lint before commit.
4. If everything passes, say so plainly and stop — do not invent findings to justify the exercise. A clean pass on a published skill is the expected result, not a suspicious one.

Judging discipline

• Quote evidence for every non-PASS verdict. No quote, no finding.
• When a verdict FAILs, first check the test question against the skill's intended design: a scenario criterion stricter than the design produces a false FAIL (real case: "opening must contain NO proposals" flunked a skill whose sanctioned opening is summary-plus-first-item). That's still a finding — but the patch is usually to make the SKILL.md state the intended behavior unambiguously, not to change the behavior.
• Judge the transcript, not the SKILL.md prose. A beautifully-written rule that didn't fire when tested is a FAIL, not a PASS-because-the-text-says-so.
• One systemic failure beats five cosmetic findings. If the same root cause (e.g. "dwell rule stated once, buried mid-file") produces failures across scenarios, report it once as the root cause with one patch.
• Remember the audience: end users of asb-* skills are strangers with no corpus. Any moment where the transcript only works because the wielder knew something outside the file is a self-containment FAIL.