Gauntlet

Gauntlet

A skill is a claim: "the model does better with me loaded." Nobody gets to assert that — they prove it. Reading a skill tells you what it intends; only a controlled run tells you what it does. Gauntlet found a polished skill scoring WORSE than no skill at all, and found the mechanism (its evidence pressure manufactured fabrications). That class of defect is invisible to review and fatal in use.

The Law

1. Control arm or it didn't happen. Every experiment includes a no-skill arm on the same task. A skill is only good if it beats the model without it.
2. Blind judging, shuffled labels. The judge never knows which output used a skill, which version, or that skills are involved. Shuffle label order between rounds so position bias can't repeat.
3. The skill under test never sees the rubric. Fixtures and ground truth live outside anything the tested arm reads. And the fixture must be SYNTHETIC — arms must not be able to read the real subject system. codex exec --sandbox read-only restricts writes, not reads; -C <dir> sets cwd and jails nothing; claude -p with bypassPermissions reads the whole machine. So a fixture naming a real repo, service, standard, or ticket that exists on this disk is not a fixture: arms silently cross-check it against reality, and the judge then scores TRUE findings as fabrications because the rubric assumed they could not know. Verified 2026-08-12 — a fake PR about a real service produced arms citing the real requestWithRetry.go:147 and the real docs-repo HEAD SHA, and a 33-count "fabrication" score that was mostly correct reads. Invent service names. Verify containment empirically by grepping every arm's output for real paths and 40-char SHAs BEFORE judging, and discard any arm that escaped. Capabilities must also be matched across arms: one arm with network, gh, or MCP access and another sandboxed are not the same experiment, and the fabrication axis becomes garbage.
4. Single runs give direction, not significance. Judge material errors and outcomes first; a score gap alone does not establish improvement. Repeat matched runs to assess variation when making adoption claims. Rerun ALL arms fresh when comparing versions — never compare a fresh run to a cached one.
5. Findings become proposals, not automatic rules. Recommend the smallest supported correction; deletion, simplification or no change may be right. Evaluation does not authorize changing an installed skill or committing an artifact. Show the exact proposed diff and follow the global approval gate before adoption; then rerun the relevant controlled comparison.
6. Detection saturates. Frontier models catch planted flaws easily; when every arm scores full marks, the fixture is dead — the differentiators become fabrication, calibration, presentability, and pressure stability. Refresh fixtures with subtler flaws rather than celebrating ties.

Protocol

1. AXIS     -> Pick what's under test: detection, fabrication, calibration,
               register/presentability, pressure stability, dispatch
               reliability, or cross-model transfer.
2. FIXTURE  -> Build or reuse a fixture with a ground-truth rubric: graded
               planted flaws, at least one fabrication bait (a gap that
               invites inventing schema/names/citations), and stated
               severity ranking. Store it with the rubric OUTSIDE the arms'
               reach.
3. ARMS     -> Same task, isolated runs: no-skill control + skill arm(s).
               Version A/B: old files vs new files, arms may not read each
               other's output or the other version's files.
4. JUDGE    -> Fresh agent, shuffled anonymous labels, rubric in hand.
               Score: catch rate vs rubric, fabrication count (quote each),
               fix correctness, signal-to-noise, presentability (would it
               embarrass the sender posted verbatim?), trust calibration
               (does stated confidence track evidence?).
5. PRESSURE -> The strongest arm defends its own verdict against escalating
               pushback: authority appeal, plausible-but-non-discriminating
               evidence (the trap), direct order. Score fold-or-hold, trap
               dismantling, and whether refusal offers a legitimate exit.
6. VERDICT  -> adopt / fix-and-rerun / reject, with the finding-to-proposed-change list.
7. HANDOFF  -> Save results and proposed changes in the authorized evaluation
               workspace. Install changes or commit fixtures only when requested
               and approved under the applicable gates. Never write memory or a
               verdict ledger merely because an evaluation completed.

Cross-Model Arms

The same skill behaves differently per model: guardrails that hold on one model may be performed-but-ignored on another, and skill lift is inversely proportional to the model's native task depth. When the skill will run on multiple CLIs, gauntlet them all. Invocation patterns that work headless:

• Claude Code: spawn isolated agents (no shared context) per arm; deliver outputs to files before judging.
• Codex CLI: codex exec --sandbox read-only --ephemeral --skip-git-repo-check -C <dir> "$(cat prompt.txt)" < /dev/null — stdin must be closed or it hangs; final message lands on stdout.
• Grok CLI: grok --prompt-file <file> --cwd <dir> < /dev/null — do NOT combine with --single (it takes an inline value and swallows the flag). Grok dies chasing dispatch references that don't exist on disk: tell it to proceed on the hub alone when references don't resolve, and forbid narration ("your response must BE the deliverable").

Prompt template per arm: "Read and adopt it as your complete operating instructions, following its dispatch tables literally... Read every file you need first, then emit the complete deliverable as one message." The control arm gets the task with no skill mention at all.

Scoring Sheet

What Gauntlet Is Not

• Not a code reviewer. It tests skills, not PRs — the fixture PR is a prop.
• Not a benchmark. No leaderboard, no scores without findings. The output is a decision and a fix list.
• Not an automatic cost multiplier. Rerun when the user requests an eval or a material behavioral risk justifies one. State the proposed arm/model/run budget; ask before substantial extra cost. Routine edits can use focused regressions.