gabrielmoreira Model Selection Advisor Skill
The right model is rarely "the biggest one" or "the cheapest one" — it's the smallest model that clears the task's quality bar within its latency and cost budget, with a path to escalate the hard cases. This skill makes that trade-off explicit and defensible, and ties it to an eval so the choice is measured, not vibes.
Working from a brief
Given "what model should I use for summarising support tickets?", deliver a concrete recommendation anyway — infer the task's difficulty, volume, and latency sensitivity, label the assumptions, and recommend. Never hand back "it depends" with no pick; give a default and the condition under which you'd change it.
Required Inputs
Ask for these only if they aren't already provided (else infer and label):
- The task — what the model does, and an example input/output. How hard is it (extraction vs. reasoning vs. open-ended)?
- Quality bar — what "good enough" means, and the cost of a wrong answer.
- Volume & latency — requests/day and how fast a response must come back (interactive vs. batch).
- Constraints — budget, context-length needs, tool use, privacy/region, and whether outputs must be reproducible.
Output Format
Model Recommendation: [task]
1. Decision criteria — the 3–5 factors that actually decide it here, ranked (e.g. reasoning depth > latency > cost), with why.
2. Option comparison — the realistic candidates scored against the criteria. Keep it provider-agnostic in method; name a default family (e.g. the Claude family — a small/fast tier, a balanced tier, a frontier tier) and reason by tier, not a single hardcoded model, so the advice survives model releases.
| Option (tier) | Quality on this task | Latency | Relative cost | Fit |
|---|---|---|---|---|
| Small/fast | clears bar for easy cases | low | $ | default for the bulk |
| Balanced | clears bar for most cases | med | $$ | when small misses |
| Frontier | clears the hardest cases | higher | $$$ | escalation / eval judge |
3. Recommendation — the default model/tier, in one sentence, with the single reason.
4. Routing strategy — cheap-by-default with escalation: run the small tier first, detect low-confidence or hard cases (length, ambiguity, a validator/judge failing), and escalate those to a stronger tier. This usually beats picking one model for everything on both cost and quality.
5. Validation — how to confirm the choice: a small eval set scored per tier (pair with
eval-rubric-designer and ai-eval-plan),
and a cost/latency estimate at real volume (pair with llm-cost-latency-budget).
Quality Checks
- The recommendation names a default model/tier and the condition that would change it
- Reasoning is by tier (small/balanced/frontier), not a single hardcoded model that dates quickly
- A routing/escalation strategy is considered, not just a single fixed choice
- The choice is tied to a measurable quality bar and an eval to verify it
- Cost and latency are estimated at real volume, not per single call
- Constraints (context length, privacy/region, reproducibility, tool use) are checked against the pick
Anti-Patterns
- Do not default to the biggest model "to be safe" — pay only for the capability the task needs
- Do not pick on price alone — a cheap model that fails the bar costs more in rework and trust
- Do not recommend without an eval to confirm the quality bar is actually met
- Do not hardcode a single model name as the answer — reason by tier and let the eval pick the current best in it
- Do not ignore the long tail — design for the hard cases via escalation, not by oversizing everything
Based On
Model-selection practice — quality/cost/latency trade-offs, tiered routing with escalation, and eval-driven validation.