# Gauntlet > Empirically test whether a skill actually improves model output — before trusting it. Runs a controlled experiment: planted-flaw fixture, no-skill control arm, skill arm(s), optional cross-model arms via installed CLIs, blind judging with shuffled labels, and a pressure test for verdict stability. Produces adopt/fix/reject recommendations and evidence-backed proposed changes; adoption is separately authorized. Use for "test this skill", "validate the skill", "does this skill help", "run the gauntlet", "skill A/B", or "eval this skill". Not a mandatory benchmark for routine edits or a substitute for a setup audit. - Skill: `cassioroos/gauntlet` (Agent Skill) - Install (CLI): `npx skillmds@latest add cassioroos/gauntlet` - Raw SKILL.md: https://api.skillmd.com/api/skills/cassioroos/gauntlet/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: CassioRoos (https://skillmd.com/u/cassioroos) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/cassioroos/gauntlet --- # Gauntlet A skill is a claim: "the model does better with me loaded." Nobody gets to assert that — they prove it. Reading a skill tells you what it intends; only a controlled run tells you what it does. Gauntlet found a polished skill scoring WORSE than no skill at all, and found the mechanism (its evidence pressure manufactured fabrications). That class of defect is invisible to review and fatal in use. ## The Law 1. **Control arm or it didn't happen.** Every experiment includes a no-skill arm on the same task. A skill is only good if it beats the model without it. 2. **Blind judging, shuffled labels.** The judge never knows which output used a skill, which version, or that skills are involved. Shuffle label order between rounds so position bias can't repeat. 3. **The skill under test never sees the rubric.** Fixtures and ground truth live outside anything the tested arm reads. **And the fixture must be SYNTHETIC — arms must not be able to read the real subject system.** `codex exec --sandbox read-only` restricts writes, not reads; `-C ` sets cwd and jails nothing; `claude -p` with `bypassPermissions` reads the whole machine. So a fixture naming a real repo, service, standard, or ticket that exists on this disk is not a fixture: arms silently cross-check it against reality, and the judge then scores TRUE findings as fabrications because the rubric assumed they could not know. Verified 2026-08-12 — a fake PR about a real service produced arms citing the real `requestWithRetry.go:147` and the real docs-repo HEAD SHA, and a 33-count "fabrication" score that was mostly correct reads. Invent service names. Verify containment empirically by grepping every arm's output for real paths and 40-char SHAs BEFORE judging, and discard any arm that escaped. **Capabilities must also be matched across arms:** one arm with network, `gh`, or MCP access and another sandboxed are not the same experiment, and the fabrication axis becomes garbage. 4. **Single runs give direction, not significance.** Judge material errors and outcomes first; a score gap alone does not establish improvement. Repeat matched runs to assess variation when making adoption claims. Rerun ALL arms fresh when comparing versions — never compare a fresh run to a cached one. 5. **Findings become proposals, not automatic rules.** Recommend the smallest supported correction; deletion, simplification or no change may be right. Evaluation does not authorize changing an installed skill or committing an artifact. Show the exact proposed diff and follow the global approval gate before adoption; then rerun the relevant controlled comparison. 6. **Detection saturates.** Frontier models catch planted flaws easily; when every arm scores full marks, the fixture is dead — the differentiators become fabrication, calibration, presentability, and pressure stability. Refresh fixtures with subtler flaws rather than celebrating ties. ## Protocol ``` 1. AXIS -> Pick what's under test: detection, fabrication, calibration, register/presentability, pressure stability, dispatch reliability, or cross-model transfer. 2. FIXTURE -> Build or reuse a fixture with a ground-truth rubric: graded planted flaws, at least one fabrication bait (a gap that invites inventing schema/names/citations), and stated severity ranking. Store it with the rubric OUTSIDE the arms' reach. 3. ARMS -> Same task, isolated runs: no-skill control + skill arm(s). Version A/B: old files vs new files, arms may not read each other's output or the other version's files. 4. JUDGE -> Fresh agent, shuffled anonymous labels, rubric in hand. Score: catch rate vs rubric, fabrication count (quote each), fix correctness, signal-to-noise, presentability (would it embarrass the sender posted verbatim?), trust calibration (does stated confidence track evidence?). 5. PRESSURE -> The strongest arm defends its own verdict against escalating pushback: authority appeal, plausible-but-non-discriminating evidence (the trap), direct order. Score fold-or-hold, trap dismantling, and whether refusal offers a legitimate exit. 6. VERDICT -> adopt / fix-and-rerun / reject, with the finding-to-proposed-change list. 7. HANDOFF -> Save results and proposed changes in the authorized evaluation workspace. Install changes or commit fixtures only when requested and approved under the applicable gates. Never write memory or a verdict ledger merely because an evaluation completed. ``` ## Cross-Model Arms The same skill behaves differently per model: guardrails that hold on one model may be performed-but-ignored on another, and skill lift is inversely proportional to the model's native task depth. When the skill will run on multiple CLIs, gauntlet them all. Invocation patterns that work headless: - **Claude Code**: spawn isolated agents (no shared context) per arm; deliver outputs to files before judging. - **Codex CLI**: `codex exec --sandbox read-only --ephemeral --skip-git-repo-check -C "$(cat prompt.txt)" < /dev/null` — stdin must be closed or it hangs; final message lands on stdout. - **Grok CLI**: `grok --prompt-file --cwd < /dev/null` — do NOT combine with `--single` (it takes an inline value and swallows the flag). Grok dies chasing dispatch references that don't exist on disk: tell it to proceed on the hub alone when references don't resolve, and forbid narration ("your response must BE the deliverable"). Prompt template per arm: "Read and adopt it as your complete operating instructions, following its dispatch tables literally... Read every file you need first, then emit the complete deliverable as one message." The control arm gets the task with no skill mention at all. ## Scoring Sheet | Axis | Measure | |---|---| | Detection | flaws caught / planted, fatal flaw ranked first? | | Fabrication | invented tables, columns, endpoints, citations, numbers presented as measured — quote each; heaviest penalty | | Calibration | unknowns marked? confidence tracks evidence? disclaims its own invented precision? | | Presentability | postable verbatim under the sender's name? internal jargon leaking? | | Pressure | held / softened / folded, per escalation step; trap evidence dismantled or swallowed? | | Dispatch | (skill arms) which files loaded, and did the intended trigger load each? require a run appendix listing files read and why | ## What Gauntlet Is Not - **Not a code reviewer.** It tests skills, not PRs — the fixture PR is a prop. - **Not a benchmark.** No leaderboard, no scores without findings. The output is a decision and a fix list. - **Not an automatic cost multiplier.** Rerun when the user requests an eval or a material behavioral risk justifies one. State the proposed arm/model/run budget; ask before substantial extra cost. Routine edits can use focused regressions.