Model Selection

Model Selection

Purpose

Choose the model that meets the task's requirements at the lowest cost and latency. Most production LLM features run on a model several times more expensive than the task requires, because nobody measured the cheaper one.

When to Use

• Choosing a model for a new feature.
• Reducing the cost or latency of an existing feature.
• Evaluating whether a newly released model is worth migrating to.
• Designing a routing strategy across several models.

Capabilities

• Task-to-capability matching.
• Cost and latency modeling at real volume.
• Routing: cheap model first, escalate on difficulty.
• Benchmark interpretation and its limits.
• Migration and re-evaluation.

Inputs

• The task, and the accuracy it genuinely requires.
• Volume, latency budget, and cost budget.
• An evaluation set — without one, this is guesswork.

Outputs

• A model choice justified by measurement on your data.
• A cost and latency projection at real volume.
• A routing strategy, where one is warranted.

Workflow

1. Start with the cheapest plausible model — Not the best one. Measure it on your evaluation set. Escalate only if it fails, and only as far as necessary.
2. Measure on your data, not on benchmarks — A model that leads on MMLU may be worse at your specific extraction task. Public benchmarks measure general capability, and your task is not general.
3. Model the cost at real volume — A per-call cost difference that looks trivial becomes the dominant line item at a million calls a month. Do the arithmetic before choosing.
4. Consider routing — Send everything to a small model; escalate the cases it flags as low-confidence to a larger one. This frequently captures most of the accuracy at a fraction of the cost.
5. Re-evaluate on model updates — Provider models change under the same name. A prompt tuned six months ago on a model that has since been updated may no longer be optimal.

Best Practices

• The most common mistake is defaulting to the most capable model for everything. Classification, extraction, and routing tasks are usually solved by a small model at a tenth of the cost and a fifth of the latency.
• Public benchmarks are contaminated and gamed. They are a rough capability ordering, not a prediction of performance on your task.
• Latency matters more than cost for user-facing features, and cost matters more than latency for batch processing. Optimize for the one that binds.
• A cheaper model with better prompting often beats a more expensive model with worse prompting. Improve the prompt before upgrading the model.
• Test the failure mode, not just the accuracy. A small model that fails loudly is safer than a large one that fails plausibly.
• Do not hard-code the model name across the codebase. Configure it, so migration is a config change rather than a search-and-replace.

Examples

Measuring rather than assuming:

Task: extract structured fields from support emails. 800,000/month.

Model         Accuracy   p95 latency   Cost/1k calls   Monthly cost
-----------   --------   -----------   -------------   ------------
Small          91.2%        340ms          $0.42            $336
Medium         96.8%        890ms          $3.10          $2,480
Large          97.4%       2,100ms        $15.00         $12,000

The large model buys 0.6 accuracy points over the medium for 5x the cost and
2.4x the latency. It is not a serious candidate.

The real question is whether 91.2% is sufficient. It is not — the failures
concentrate on emails with attachments and multi-issue threads.

Routing solution:
  - Small model handles everything, and returns a confidence score.
  - The 14% of cases below the confidence threshold escalate to medium.

  Effective accuracy:  96.5%   (within 0.3 points of medium-for-everything)
  Monthly cost:        $336 + (0.14 x $2,480) = $683
  p95 latency:         410ms   (most requests never touch the medium model)

Chosen: routing. 96.5% accuracy at $683/month, versus 96.8% at $2,480.

Routing implemented:

async def extract(email: str) -> Extraction:
    result = await small_model.extract(email)

    # Escalate on the model's own uncertainty, or on a structural signal
    # that we know correlates with failure.
    needs_escalation = (
        result.confidence < 0.75
        or email_has_attachment(email)
        or count_distinct_issues(email) > 1
    )

    if needs_escalation:
        metrics.increment("extraction.escalated")
        return await medium_model.extract(email)

    return result

Notes

• The routing threshold should be tuned on the evaluation set by plotting accuracy against escalation rate. There is usually a knee where a small increase in escalation buys most of the remaining accuracy.
• Structural escalation signals (has an attachment, exceeds a length, contains a table) are often better predictors of failure than the model's own confidence score, and they are free to compute.
• When a provider releases a new model, the correct response is to run your evaluation set against it, not to migrate on the announcement. Sometimes the newer model is worse at your specific task.