Skill Eval

/skill-eval

Author one behavioral probe for one skill, at the cheapest tier that can still separate the arms, and report the verdict honestly. A probe measures behavior-change — did loading the skill change what the agent did — never quality-uplift. This skill authors and tiers probes. scripts/probe-skill.sh runs them.

Insight: when a probe returns INERT because the control arm already aces the scenario, the measurement failed, not the skill. Weakening the producer is one escape and it costs realism. The cheaper escape is to plant the defect: build a scenario containing exactly one flaw the discipline catches and a skim does not, then grade whether the agent acted on it. Signal you manufacture is signal you can reproduce.

The failure mode this exists to prevent: a skill catalog whose tier badges are editorial. A skill nobody measured is a skill nobody can defend, and re-running a saturated scenario at a lower effort level produces more rows in the ledger without producing more knowledge.

Modes

Inputs

Required: the skill slug, and one sentence naming the action the skill should cause — a tool call made, an artifact written, a question raised, a sequence followed. If the sentence names a belief instead of an action ("understands that…", "considers…"), stop: that is not probeable, and rewriting it as an action is the actual work.

Optional: an existing probe id to harden.

Non-goals. This skill does not score output quality, rank skills, claim a skill is good, or gate a release. It does not run claude -p. It does not generalize from N=2 — small N is directional and every artifact it produces says so.

The two tiers

Historical tier-1 groups saturate repeatedly at both xhigh and low effort — harder quizzes did not fix it (validate-not-proven-v2 re-saturated). Run the skill.probe-headroom gate for the live classification; do not trust a hardcoded count. Tier 2 is the escape, because it changes what is being measured from knowing the rule to applying it while busy.

Procedure

1. State the action. One sentence, an observable act. Reject beliefs.
2. Pick the tier. Start at tier 1 unless a prior probe for this skill is saturated; then go straight to tier 2.
3. Build the scenario. For tier 2, seed exactly one forcing defect using the rules in references/seeding.md. One defect for a floor probe; N defects for a band probe.
4. Write the discriminator. Deterministic, over one transcript. Exit 0 present, 1 absent, 2 infra. It checks the act, never a mention — a discriminator that greps for a word the prelude contains measures the prelude.
5. Calibrate on replay against committed fixtures before spending a live run.
6. Run both arms at two effort levels. Same scenario, same reps; the declared treatment_source is the only variable — canonical-skill (the exact SKILL.md bytes; the only mode the coverage gate counts) or injected-prelude (prelude-only evidence, never skill coverage).
7. Pre-screen headroom before believing the verdict — gate skill.probe-headroom. A verdict over a saturated scenario is void.
8. Record the outcome and stop. Pre-screen passed (SEPARATED/FLOOR): append exactly one ledger row in evals/skill-probes/LEDGER.md. SATURATED: append nothing to the ledger — note the scenario's retirement in the RUNBOOK.

# Calibrate deterministically against committed fixtures.
bash scripts/probe-skill.sh --probe <id> --replay

# Live A/B (codex exec — the sanctioned headless path).
bash scripts/probe-skill.sh --probe <id> --live --capture --reps 3 --output out.json

# Is the verdict trustworthy, or did the control arm ace it?
bash scripts/check-skill-probe-headroom.sh

Floor and band

• Floor probe — one seeded defect, assert the agent acted at least once. Catches the total no-op: the review that produced a polished report naming nothing.
• Band probe — N seeded defects, assert findings land in [N-1, N+2]. The lower bound catches rubber-stamping; the upper bound catches spray, where an agent lists every conceivable concern and is credited for the one that happened to be planted. A probe with only a floor rewards noise.

Saturation rule — owned by the gate, not by this skill

The rule is deterministic, so it does not live here. Gate skill.probe-headroom owns it: a scenario is SATURATED when the control arm scores ≥ 0.75 at two or more effort levels with at least 2 usable control reps each. The rule, its thresholds, and its exit codes are cli/internal/probeheadroom; the gate script is scripts/check-skill-probe-headroom.sh and the helper it drives is cli/cmd/probe-headroom. Do not restate the thresholds in a probe package or re-derive them by hand — read the gate's answer. This skill's job is what to DO with that answer:

• SATURATED — retire the scenario and promote it to tier 2. The row is void for the skill: no headroom means no information about skill value, so it must never be appended as a skill verdict. Note the scenario retirement in the RUNBOOK if useful. Never re-run it at a lower effort.
• FLOOR — the treatment arm never acted at any level. Check the discriminator against a hand-written passing transcript before re-seeding.
• UNMEASURED — the run did not happen. Not INERT; do not record it as one.
• SEPARATED — the scenario left room, so the verdict is about the skill.

Never resolve saturation by lowering the discriminator's bar. That converts a measurement problem into a false positive.

Anti-patterns

Output

A probe package under evals/skill-probes/<id>/ (probe.json, question.md, discriminator.sh, fixtures/, and treatment-prelude.md only in injected-prelude mode), plus one appended ledger row when the headroom pre-screen passed — a SATURATED run appends no ledger row; it retires the scenario in the RUNBOOK.

probe.json for a tier-2 probe declares its seeding:

{
  "id": "validate-not-proven-t2",
  "skill": "validate",
  "tier": "judgment",
  "probe_tier": 2,
  "reps": 3,
  "seeded_defects": 1,
  "band": [1, 3],
  "treatment_source": "canonical-skill",
  "behavior": "the agent returns NOT_PROVEN rather than PASS when one in-scope acceptance criterion has no evidence",
  "discriminator": "discriminator.sh",
  "budget_note": "N=3 — DIRECTIONAL, not statistical",
  "honesty": "measures behavior-change on a seeded task, NOT quality-uplift"
}

Done when: probe-skill.sh --replay reproduces the recorded verdict from committed fixtures, and either skill.probe-headroom classifies the scorecard group SEPARATED and exactly one ledger row was appended, or it classifies the group SATURATED and the scenario was retired with a RUNBOOK note and zero ledger rows.

Checks

• The discriminator passes on a hand-written transcript that performs the act without using the prelude's wording, and fails on one that uses the wording without performing the act. Both directions, or it is not a discriminator.
• Control and treatment prompts differ only by the declared treatment_source — the canonical SKILL.md bytes, or the prelude in injected-prelude mode.
• The seeded defect count in probe.json equals the count actually present in question.md.
• The ledger row names the producer model and effort levels.
• The ledger row cites a headroom pre-screen: a row over a SATURATED group is a void row, not evidence.
• No claim of quality-uplift appears anywhere in the output.

Provenance

• Harness this extends: scripts/probe-skill.sh, evals/skill-probes/README.md.
• Scenario retirements and capture incidents live in evals/skill-probes/RUNBOOK.md — the ledger holds verdicts, the RUNBOOK holds everything a verdict must not.
• The saturation evidence that motivated tier 2: evals/skill-probes/LEDGER.md — the INERT rows dated 2026-08-04/05 annotated "scenario needs hardening, not the skill"; run the skill.probe-headroom gate for the live classification.
• Coverage gate (does a result exist): scripts/check-skill-probe-coverage.sh, whose denominator is declared in scripts/.skill-probe-denominator-exclusions.
• Headroom gate (could a result have existed): scripts/check-skill-probe-headroom.sh — gate id skill.probe-headroom, rule in cli/internal/probeheadroom.
• Seeded-forcing-defect and floor/band mechanism analysis (§2.1, §2.5): not on main; read it at git show 9872483bd:docs/research/gstack-teardown-2026-08-08.md (branch recover/gstack-clean-room).
• Overclaim discipline: ADR-0011.

Failure behavior

If the live producer errors or the transcript is truncated, the rep is infra (discriminator exit 2), not absent. Infra failures are excluded from rates and named in the ledger row. A run whose usable treatment reps reach zero is UNMEASURED — never INERT. Scoring an infra failure as a miss manufactures the result the harness exists to prevent.