Evaluate Matrix

/evaluate:matrix

The executability gate. Where /evaluate:legibility asks "can a fresh agent comprehend this skill," this skill asks "can a weak model actually do it" — run the skill's evals on haiku (and opus/sonnet) with real tool execution, grade the produced artifact, and surface the per-skill verdict executable_on_haiku. A skill that opus passes and haiku fails leans on reasoning the cheap model lacks.

This builds the orchestration the cross-model design (docs/cross-model-evaluation.md) calls a follow-up. It reuses, without duplicating: prepare_run.sh, grade_deterministic.py (zero-token first pass), the eval-grader agent (deferred fuzzy checks only), model-matrix.json, and render_matrix_report.py.

When to Use This Skill

Context

• Available plugins: !find . -maxdepth 2 -type d -name '*-plugin' -not -name '.claude-plugin'

Parameters

Aliases float across model generations, so --models accepts either an alias or a full id; pass the full id when the run must be reproducible (the model field on the dispatch accepts one) and record the id that actually ran — never the alias — in model-matrix.json metadata.models[].model_id (.claude/rules/skill-evaluation.md).

Execution

Execute this cross-model matrix:

Step 1: Resolve skill and evals

Read <plugin-name>/skills/<skill-name>/evals.json. If absent, report that the matrix needs eval cases (point at /evaluate:skill --create-evals) and stop. Validate it against the evals.json schema (references/schemas.md).

Step 2: Run the matrix (serialized)

Loop over (model ∈ --models) × eval × config ∈ {with_skill, cached_baseline} — skip cached_baseline if --with-skill-only, and reuse a baseline cached for the same model-version rather than re-running it. For each combination:

1. Scaffold the run dir:

   bash ${CLAUDE_PLUGIN_ROOT}/scripts/prepare_run.sh \
     --skill-dir <plugin-name>/skills/<skill-name> \
     --eval-id <eval-id> --run <N>
   

   If the eval carries a fixture block, apply it for an honest execution context — without one a context-needing skill fails on haiku purely for lack of fixtures, a false negative that poisons this gate:

   bash ${CLAUDE_PLUGIN_ROOT}/scripts/apply_fixture.sh \
     --fixture '<eval.fixture JSON>' --repo-root "$(pwd)"
   

   Parse WORKDIR=; the subagent operates there. Tear it down after the transcript is copied out (--teardown "$WORKDIR").
2. Dispatch one Task subagent with the model field set to the loop model (full Bash/Edit — it does real tool execution, not just reading):

   Task subagent_type: general-purpose
   model: <loop model alias>
   prompt: <eval prompt; with_skill runs also receive the SKILL.md content>
   

   Serialize the dispatches — one at a time, never a parallel batch. [1m] models hit cascading rate limits with concurrent subagents (.claude/rules/skill-fork-context.md). Spawns run in the background by default (Claude Code ≥ 2.1.232), so dispatch order alone no longer serializes them: pass run_in_background: false on each dispatch, or wait for that agent's result to arrive before issuing the next one — serialization is the wait, not the dispatch order.
3. Write the subagent's produced artifact to $RUN_DIR/transcript.md.

Step 3: Grade — deterministic first, judge only on deferral

For each run, grade the produced output:

1. Run the zero-token deterministic grader first:

   python3 ${CLAUDE_PLUGIN_ROOT}/scripts/grade_deterministic.py \
     --evals <evals.json> --eval-id <eval-id> --output $RUN_DIR/transcript.md --json
   

2. Only if it reports JUDGE_PENDING > 0, dispatch the eval-grader agent for the deferred fuzzy expectations:

   Task subagent_type: evaluate-plugin:eval-grader
   Prompt: Grade ONLY the deferred (judge) expectations for <eval-id> ...
   

   Most expectations grade deterministically — the judge fires on a fraction.

Step 4: Aggregate to model-matrix.json

Combine per-run pass rates into <skill-dir>/eval-results/model-matrix.json following the schema. Compute, per model alias, the mean with_skill and baseline, the delta, and prev_delta from any stored prior run.

Step 5: Render the report

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/render_matrix_report.py \
  <skill-dir>/eval-results/model-matrix.json

The renderer emits the delta table, per-model verdicts, the portability flag (opus−haiku spread ≥20 pts), and the executability flag (executable_on_haiku=false when haiku's absolute with-skill rate is below the 0.5 floor while opus clears it). Print the report and call out whether the executability flag fired.

Minimal Provable Increment

Run the matrix for git-plugin/git-commit only (it already has typed-check evals), --models opus,haiku --with-skill-only: grade deterministically, render, and confirm the executability callout lights up or stays dark correctly. This exercises every reused piece end-to-end before scaling to the golden set.

Agentic Optimizations

Quick Reference

Related

• /evaluate:legibility — the comprehension gate (cold-read, no execution)
• /evaluate:skill — single-model effectiveness with a baseline
• docs/cross-model-evaluation.md — the design this implements
• .claude/rules/skill-evaluation.md — tiered methodology, golden set, cadence
• .claude/rules/skill-fork-context.md — why subagent dispatch is serialized