Model Ab Test

Model A/B Test Harness

Run each test case from a definition file: dispatch two subagents (baseline vs proposed model) in parallel, score outputs binary per criterion, write evidence files, produce verdict.

Phase 1: Parse Test Definitions

Split $ARGUMENTS — first token = file path, second (optional) = test name filter (case-insensitive substring match on ### <name> headings).

Read the file. For each test case block, extract:

Skip cases with missing fields (log warning). Skip ~~strikethrough~~ or EXEMPTED headings entirely.

Phase 2: Locate Content + Preloaded Skills

For each test case name:

1. Glob plugins/*/skills/{name}/SKILL.md — if found, this is primary_content (type: skill)
2. Else glob plugins/*/agents/{name}.md — if found, this is primary_content (type: agent)
3. If agent has skills: in frontmatter, parse comma-separated names. For each:
   • If qualified (plugin:skill), split on : → glob plugins/{plugin}/skills/{skill}/SKILL.md
   • If bare name → glob plugins/*/skills/{name}/SKILL.md
   • Read content, store in preloaded_skills map
4. Not found → log error, skip test case

Phase 3: Dispatch A/B Subagents

For each test case, assemble the prompt using the template from references/subagent-prompt.md. Replace placeholders with resolved content.

Dispatch TWO Task subagents in a SINGLE message (parallel):

Task A: subagent_type="general-purpose", model="opus"
Task B: subagent_type="general-purpose", model="{proposed_model}"

Both receive the identical assembled prompt. No modifications for either model.

After both return:

• Write Task A output → _docs/model-tests/results/{name}-opus.md
• Write Task B output → _docs/model-tests/results/{name}-{proposed_model}.md

Phase 4: Score Results

Read both output files. Score each criterion binary — no "close enough":

GREEN criteria — for each, per model: pass or fail RED indicators — for each, per model: clean or triggered

Every score MUST cite a specific quote from the output as evidence.

Verdict logic:

Write comparison entry per test case:

## {name} (Opus -> {proposed_model})

### GREEN Criteria
| # | Criterion | Opus | {proposed_model} |
|---|-----------|:----:|:----------------:|
| 1 | {text} | pass/fail | pass/fail |

### RED Indicators
| # | Indicator | Opus | {proposed_model} |
|---|-----------|:----:|:----------------:|
| 1 | {text} | clean/triggered | clean/triggered |

### Verdict: {PASS/FAIL/INCONCLUSIVE}
{1-2 sentences citing specific evidence from outputs.}

Phase 5: Summary Report

After all test cases, derive the report filename from the source test definition file: strip path and extension to get the stem (e.g., 01-haiku-templates from _docs/model-tests/01-haiku-templates.md). Write _docs/model-tests/results/{stem}-report.md:

# Model A/B Comparison Report
Generated: {date} | Source: {test definition file}

{all comparison entries from Phase 4}

---

## Summary
| Skill/Agent | Downgrade | GREEN | RED | Verdict |
|-------------|-----------|:-----:|:---:|:-------:|
| {name} | Opus->{model} | {n}/{total} | {n} new | PASS/FAIL/INCONCLUSIVE |

## Safe to Downgrade
{PASS verdicts with justification, or "None."}

## Keep on Opus
{FAIL verdicts with failed criteria, or "None."}

## Needs More Testing
{INCONCLUSIVE verdicts with what would resolve, or "None."}

Sequencing

• Test cases within a file: sequential (one at a time)
• Subagents A and B for same test case: parallel (both Task calls in one message)

Discipline Rules

These rules exist because without them, agents rationalize shortcuts under pressure. Each rule addresses a documented failure mode.

Common Mistakes