LLM Model Selection — Consult Benchmarks Before Choosing
When a task requires you to recommend or select an LLM for a specific use case, follow a systematic process before defaulting to a well-known model name:
1. Check Existing Benchmarks First
If you or your team have already run benchmarks (bake-offs) for similar use cases, that prior data is more valuable than generic leaderboards. Reasons:
- Context-specific results trump averages. A model that ranks #2 on MMLU may rank #1 on your specific task (code generation, legal analysis, math reasoning).
- Real-world performance differs from benchmark suites. Generic leaderboards test broad knowledge; your task has specific constraints (latency budget, cost per call, long-context handling, instruction-following precision).
- You avoid redundant work. If you've already benchmarked "agent reasoning on financial data," reuse that result instead of re-running it on a similar financial-domain task.
If an existing benchmark covers your use case: use the winner as your default choice. Note the selection criteria so you can explain the decision.
2. When to Re-Benchmark
If your use case diverges from existing benchmarks, narrow the test scope and run a fresh benchmark:
- Scope mismatch: existing benchmark was for "summarization" but your task is "multi-step reasoning with chain-of-thought"
- Different constraints: existing test was on 4K-token documents; you need 100K-token handling
- New model release: new frontier models are available since the last benchmark
- Cost/latency window changed: you now have stricter latency budget or cost ceiling
When re-benchmarking, use existing benchmarks as prior signals — especially for known weaknesses (e.g., "GPT models historically fail at precise JSON output; Claude handles it better"). This lets you design a test that's smaller and faster than the original.
3. Always Validate Availability & Pricing Live
Never trust model names or pricing from memory or old docs. Models are deprecated, renamed, and repriced frequently. Always query the live API or vendor's current listing:
- Check the model catalog (OpenRouter, Claude API, Together.ai, local LLM registry, etc.) for current slugs and availability.
- Validate pricing — per-1M-token costs change, some models graduate from beta pricing.
- Confirm context window — older docs may list smaller limits than current releases.
- Check availability in your region — some models are geo-gated.
This query takes seconds and prevents selecting a model that no longer exists or is no longer cost-effective for your budget.
4. Generic Leaderboards as a Last Resort
If you have no prior benchmarks and no time to run one, generic leaderboards (Artificial Analysis, Vellum, HELM, MMLU rankings) are a starting point, not the final answer. Use them to identify promising candidates, then ask yourself:
- Does this leaderboard test my actual constraints? (e.g., a leaderboard for "general knowledge" doesn't predict performance on specialized code generation)
- What does the test suite optimize for? (latency, reasoning depth, factuality, instruction-following?) Does it match your priorities?
- Are the costs/latencies published alongside the scores? A model that's 1% more accurate but 10× slower may be the wrong choice.
From the leaderboard's top candidates, pick 2–3 contenders and run a small focused benchmark (10–20 examples from your real task) before committing.
Anti-Patterns to Avoid
- Treating leaderboards as oracle: "Claude scores highest on MMLU, so it's best for my task." Not necessarily — your task may have different structure.
- Choosing based on brand or hype: "Everyone uses GPT-4, so use that." Popular ≠ optimal for your use case.
- Assuming past performance is current: Model versions change, pricing shifts, availability in your region may be different. Revalidate.
- Benchmarking once and never updating: New models release every few months. Revisit selection every 3–6 months if your task is mission-critical.
- Ignoring the "why" of a choice: If you can't explain why Model A beats Model B for your task (beyond "it scored higher on a generic test"), you've chosen prematurely.
Workflow Summary
- Do you have past benchmark data for this use case? → Use the winner; document why it was chosen.
- Does your use case differ from past benchmarks? → Design a focused benchmark for the differing part; use past results as priors.
- Do you have time/budget to benchmark? → Run a small (10–20 example) test on top candidates from leaderboards.
- Validate live: Check model availability, pricing, and context window on the current API before finalizing the choice.
- Document the decision: Write down the criteria, the benchmark (if any), and the runner-up models. This helps future decisions and justifies the choice to stakeholders.
Applies To
- All AI agent frameworks and development environments (Claude Code, Cursor, etc.).
- Any task involving LLM selection, from personal projects to production systems.
- Works for API-based models (OpenRouter, Anthropic, OpenAI, Mistral, etc.) and self-hosted models (Ollama, vLLM, local quantized runs).
1---2name: llm-model-selection3description: Select the right LLM for a task by consulting benchmarks and real performance data. Use when choosing between models (chatbot, agent, text classification, code generation, etc.), comparing model performance, or validating model availability and pricing.4---56# LLM Model Selection — Consult Benchmarks Before Choosing78When a task requires you to recommend or select an LLM for a specific use case, follow a systematic process before defaulting to a well-known model name:910## 1. Check Existing Benchmarks First1112If you or your team have already run benchmarks (bake-offs) for similar use cases, that prior data is more valuable than generic leaderboards. Reasons:1314- **Context-specific results trump averages.** A model that ranks #2 on MMLU may rank #1 on your specific task (code generation, legal analysis, math reasoning).15- **Real-world performance differs from benchmark suites.** Generic leaderboards test broad knowledge; your task has specific constraints (latency budget, cost per call, long-context handling, instruction-following precision).16- **You avoid redundant work.** If you've already benchmarked "agent reasoning on financial data," reuse that result instead of re-running it on a similar financial-domain task.1718**If an existing benchmark covers your use case:** use the winner as your default choice. Note the selection criteria so you can explain the decision.1920## 2. When to Re-Benchmark2122If your use case **diverges from existing benchmarks**, narrow the test scope and run a fresh benchmark:2324- **Scope mismatch:** existing benchmark was for "summarization" but your task is "multi-step reasoning with chain-of-thought"25- **Different constraints:** existing test was on 4K-token documents; you need 100K-token handling26- **New model release:** new frontier models are available since the last benchmark27- **Cost/latency window changed:** you now have stricter latency budget or cost ceiling2829When re-benchmarking, use existing benchmarks as **prior signals** — especially for known weaknesses (e.g., "GPT models historically fail at precise JSON output; Claude handles it better"). This lets you design a test that's smaller and faster than the original.3031## 3. Always Validate Availability & Pricing Live3233**Never trust model names or pricing from memory or old docs.** Models are deprecated, renamed, and repriced frequently. Always query the live API or vendor's current listing:3435- Check the **model catalog** (OpenRouter, Claude API, Together.ai, local LLM registry, etc.) for current slugs and availability.36- Validate **pricing** — per-1M-token costs change, some models graduate from beta pricing.37- Confirm **context window** — older docs may list smaller limits than current releases.38- Check **availability in your region** — some models are geo-gated.3940This query takes seconds and prevents selecting a model that no longer exists or is no longer cost-effective for your budget.4142## 4. Generic Leaderboards as a Last Resort4344If you have no prior benchmarks and no time to run one, generic leaderboards (Artificial Analysis, Vellum, HELM, MMLU rankings) are a starting point, **not the final answer.** Use them to identify **promising candidates**, then ask yourself:4546- **Does this leaderboard test my actual constraints?** (e.g., a leaderboard for "general knowledge" doesn't predict performance on specialized code generation)47- **What does the test suite optimize for?** (latency, reasoning depth, factuality, instruction-following?) Does it match your priorities?48- **Are the costs/latencies published alongside the scores?** A model that's 1% more accurate but 10× slower may be the wrong choice.4950From the leaderboard's top candidates, pick **2–3 contenders** and run a small focused benchmark (10–20 examples from your real task) before committing.5152## Anti-Patterns to Avoid5354- **Treating leaderboards as oracle:** "Claude scores highest on MMLU, so it's best for my task." Not necessarily — your task may have different structure.55- **Choosing based on brand or hype:** "Everyone uses GPT-4, so use that." Popular ≠ optimal for your use case.56- **Assuming past performance is current:** Model versions change, pricing shifts, availability in your region may be different. Revalidate.57- **Benchmarking once and never updating:** New models release every few months. Revisit selection every 3–6 months if your task is mission-critical.58- **Ignoring the "why" of a choice:** If you can't explain why Model A beats Model B for *your* task (beyond "it scored higher on a generic test"), you've chosen prematurely.5960## Workflow Summary61621. **Do you have past benchmark data for this use case?** → Use the winner; document why it was chosen.632. **Does your use case differ from past benchmarks?** → Design a focused benchmark for the differing part; use past results as priors.643. **Do you have time/budget to benchmark?** → Run a small (10–20 example) test on top candidates from leaderboards.654. **Validate live:** Check model availability, pricing, and context window on the current API before finalizing the choice.665. **Document the decision:** Write down the criteria, the benchmark (if any), and the runner-up models. This helps future decisions and justifies the choice to stakeholders.6768## Applies To6970- All AI agent frameworks and development environments (Claude Code, Cursor, etc.).71- Any task involving LLM selection, from personal projects to production systems.72- Works for API-based models (OpenRouter, Anthropic, OpenAI, Mistral, etc.) and self-hosted models (Ollama, vLLM, local quantized runs).
avizmarlon