How do I install the Model Bakeoff skill?

Run npx skillmds@latest add ityaadiii/model-bakeoff in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Model Bakeoff safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Model Bakeoff?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Model Bakeoff free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Model Bakeoff?

ityaadiii (@ityaadiii) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Model Bakeoff by ityaadiii — Agent Skill

Name: Model Bakeoff
Author: ityaadiii

Model Bakeoff

Compare two or more models on the same task and decide whether any observed difference is real. Use when choosing between models or providers, running a bakeoff or shootout, checking if a new model beats the current one, or interpreting benchmark results. Uses paired tests and corrects for multiple comparisons. Refuses to name a winner on noise.

ityaadiii 776bd2a 2.5 KB Updated Sep 17, 2026

File contents

Comparing models without manufacturing a winner

A sweep of 5 models across 4 workloads and 5 metrics is 100 simultaneous tests. At p < 0.05 that produces about 5 "significant" results from pure noise, and those are the ones that end up in the summary.

The refusal

Do not name a winner that does not survive correction. If nothing survives, the finding is "these models are indistinguishable on this evidence", and that is a real, useful, publishable result.

Also refuse to compare unpaired when the models saw the same items. Throwing the pairing away discards most of the available power.

Procedure

Same items, every model. If they saw different items, stop. That is a different and much weaker study, and it should be labelled as one.
Count the comparisons before you run them. Models times workloads times metrics. Write the number down. It goes in the output.
Use McNemar on the discordant pairs for accuracy-style outcomes. Items every model got right, or every model got wrong, carry no information about which is better.
Correct with Holm across the full sweep, not per workload. See holm() in lib/stats.ts.
Report the effect size next to the p-value. Significant and tiny is a real category and it usually means "do not switch".
Check the practical gates separately. Latency, cost, and failure modes disqualify models regardless of accuracy. A model that wins by 2 points at 30x the latency has not won. Say so with the measured number, do not just drop it silently.

Output contract

number of comparisons run, including the ones not reported
per comparison: discordant counts, p, Holm threshold, survives yes/no
the winner, or an explicit "no winner survives correction"
disqualifications on latency or cost, with the measurement that disqualified them
per-slice results wherever an aggregate could be hiding a reversal

What this prevents

"Model B beat Model A on the extraction workload, p = 0.03." True, correctly computed, and drawn from a sweep of 100 tests where nothing survives Holm. The sentence is accurate. The finding is not there.

---name: model-bakeoffdescription: Compare two or more models on the same task and decide whether any observed difference is real. Use when choosing between models or providers, running a bakeoff or shootout, checking if a new model beats the current one, or interpreting benchmark results. Uses paired tests and corrects for multiple comparisons. Refuses to name a winner on noise.---# Comparing models without manufacturing a winnerA sweep of 5 models across 4 workloads and 5 metrics is 100 simultaneous tests. Atp < 0.05 that produces about 5 "significant" results from pure noise, and those are theones that end up in the summary.## The refusal**Do not name a winner that does not survive correction.** If nothing survives, thefinding is "these models are indistinguishable on this evidence", and that is a real,useful, publishable result.Also refuse to compare unpaired when the models saw the same items. Throwing thepairing away discards most of the available power.## Procedure1. **Same items, every model.** If they saw different items, stop. That is a different and much weaker study, and it should be labelled as one.2. **Count the comparisons before you run them.** Models times workloads times metrics. Write the number down. It goes in the output.3. **Use McNemar on the discordant pairs** for accuracy-style outcomes. Items every model got right, or every model got wrong, carry no information about which is better.4. **Correct with Holm** across the full sweep, not per workload. See `holm()` in `lib/stats.ts`.5. **Report the effect size next to the p-value.** Significant and tiny is a real category and it usually means "do not switch".6. **Check the practical gates separately.** Latency, cost, and failure modes disqualify models regardless of accuracy. A model that wins by 2 points at 30x the latency has not won. Say so with the measured number, do not just drop it silently.## Output contract- number of comparisons run, including the ones not reported- per comparison: discordant counts, p, Holm threshold, survives yes/no- the winner, or an explicit "no winner survives correction"- disqualifications on latency or cost, with the measurement that disqualified them- per-slice results wherever an aggregate could be hiding a reversal## What this prevents"Model B beat Model A on the extraction workload, p = 0.03." True, correctly computed,and drawn from a sweep of 100 tests where nothing survives Holm. The sentence isaccurate. The finding is not there.

Frequently asked questions