# Model Bakeoff

> Compare two or more models on the same task and decide whether any observed difference is real. Use when choosing between models or providers, running a bakeoff or shootout, checking if a new model beats the current one, or interpreting benchmark results. Uses paired tests and corrects for multiple comparisons. Refuses to name a winner on noise.

- Skill: `ityaadiii/model-bakeoff` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ityaadiii/model-bakeoff`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ityaadiii/model-bakeoff/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ityaadiii (https://skillmd.com/u/ityaadiii)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ityaadiii/model-bakeoff

---


# Comparing models without manufacturing a winner

A sweep of 5 models across 4 workloads and 5 metrics is 100 simultaneous tests. At
p < 0.05 that produces about 5 "significant" results from pure noise, and those are the
ones that end up in the summary.

## The refusal

**Do not name a winner that does not survive correction.** If nothing survives, the
finding is "these models are indistinguishable on this evidence", and that is a real,
useful, publishable result.

Also refuse to compare unpaired when the models saw the same items. Throwing the
pairing away discards most of the available power.

## Procedure

1. **Same items, every model.** If they saw different items, stop. That is a different
   and much weaker study, and it should be labelled as one.
2. **Count the comparisons before you run them.** Models times workloads times metrics.
   Write the number down. It goes in the output.
3. **Use McNemar on the discordant pairs** for accuracy-style outcomes. Items every
   model got right, or every model got wrong, carry no information about which is better.
4. **Correct with Holm** across the full sweep, not per workload. See `holm()` in
   `lib/stats.ts`.
5. **Report the effect size next to the p-value.** Significant and tiny is a real
   category and it usually means "do not switch".
6. **Check the practical gates separately.** Latency, cost, and failure modes disqualify
   models regardless of accuracy. A model that wins by 2 points at 30x the latency has
   not won. Say so with the measured number, do not just drop it silently.

## Output contract

- number of comparisons run, including the ones not reported
- per comparison: discordant counts, p, Holm threshold, survives yes/no
- the winner, or an explicit "no winner survives correction"
- disqualifications on latency or cost, with the measurement that disqualified them
- per-slice results wherever an aggregate could be hiding a reversal

## What this prevents

"Model B beat Model A on the extraction workload, p = 0.03." True, correctly computed,
and drawn from a sweep of 100 tests where nothing survives Holm. The sentence is
accurate. The finding is not there.