Model Fitness

Model fitness

Model choice is a hypothesis. This skill makes it an evidence-backed one.

The failure this prevents: a model gets assigned a role once, quietly fails that kind of work three times in a row, and keeps getting assigned it because nobody was counting. The opposite failure matters too — swapping away from a good model after one bad run that was really a bad prompt.

Related: test-rigor for the evidence standard, perf-proof for measuring performance claims, harness-orchestration / omp-orchestration for actually launching lanes.

The ledger

One JSONL file: ~/.model-fitness/ledger.jsonl. One line per completed lane.

node ~/.claude/skills/model-fitness/scripts/fitness.mjs record \
  --model openai-codex/gpt-5.6-sol \
  --task-type refactor-wide \
  --label P2-eager-bundle \
  --outcome pass \
  --notes "converted 12 barrel imports, build verified"

--outcome is one of:

fail-infra is excluded from fitness scoring. A flaky endpoint is not a bad model, and conflating the two is how you end up abandoning a good model. However if a flaky endpoint is stopping a current scope of work often it's appropriate to switch to a different endpoint with a different known good model.

fail-honesty is weighted hardest. A model that fabricates a number is worse than one that fails loudly, because its failures are invisible.

Task types

Keep this vocabulary small and stable or the statistics never accumulate:

Reading the record

node ~/.claude/skills/model-fitness/scripts/fitness.mjs report
node ~/.claude/skills/model-fitness/scripts/fitness.mjs report --task-type refactor-deep
node ~/.claude/skills/model-fitness/scripts/fitness.mjs suggest --task-type refactor-deep

suggest returns the best-evidenced model for a task type, and says plainly when there is not enough evidence to have an opinion.

The swap rule

Switch when a model has ≥3 scored attempts at a task type and a success rate below 50%, and another model has a better record with at least 2 attempts. Both halves matter — the second stops you swapping into the unknown.

Below 3 attempts, do not swap on the record. One or two failures is a prompt problem more often than a model problem. Before blaming the model, check:

• Was the prompt actually unambiguous about what "done" means?
• Did it have a verification contract it could run itself?
• Was the failure fail-infra wearing another label?

Any single fail-honesty is worth acting on immediately regardless of count — not necessarily by swapping, but by tightening the prompt's reporting contract and re-running.

Known shape of the roster

Working hypotheses to be overwritten by evidence, not defended:

All on included coding plans. Do not reach for orr/* (OpenRouter, per-token) unless a plan is exhausted.

Closing the loop

After any batch of delegated lanes:

1. record one line per lane, honestly. A lane you had to fix by hand is partial, not pass.
2. report to see whether anything crossed the swap threshold.
3. If it did, re-point the role in whatever config drives selection for that project, and note in the ledger why you swapped.

The ledger is only worth what you put in it. Recording a pass for a lane you quietly repaired makes every later suggestion wrong.