Cost Quality Frontier

Cost-Quality Frontier

Most eval suites today answer the question "which model is most accurate?" The production question is harder: "which model gives me the best quality I can afford, at the latency budget my product allows?" This skill takes existing eval runs and augments each result with cost and latency, then plots the candidates on a Pareto frontier so the trade-off is visible at a glance.

Core Principle

Quality, cost, and latency are a single decision, not three. A model that is 2 points more accurate but costs 6× and adds 800ms of p95 latency is rarely the right pick. A model that is 1 point worse but cheaper and faster usually is - but only if you can see the frontier. This skill makes the frontier explicit.

What You'll Get

Augmented Eval Result Schema

The skill assumes your existing eval results are tabular (CSV / JSON / DataFrame). It extends the schema with these columns; existing columns are preserved unchanged.

A reference pricing.json ships with the skill in pricing.example.json. Maintainers update this file when prices change; every eval run records which version of pricing was used so historical results stay reproducible. Pricing is read from a local file - the skill never makes external calls to fetch prices.

{
  "_version": "2026-04-15",
  "_note": "Per-million-token prices in USD. Replace with current vendor prices before relying on results.",
  "models": {
    "claude-sonnet-4-5":  { "in_per_mtok": 3.00,  "out_per_mtok": 15.00 },
    "claude-opus-4-6":    { "in_per_mtok": 15.00, "out_per_mtok": 75.00 },
    "claude-haiku-4-5":   { "in_per_mtok": 0.80,  "out_per_mtok": 4.00  },
    "gpt-5-mini":         { "in_per_mtok": 0.25,  "out_per_mtok": 2.00  },
    "gpt-5":              { "in_per_mtok": 2.50,  "out_per_mtok": 10.00 }
  }
}

Process

Step 1: Intake

I'll ask:

"Point me at your eval results (file or paste a sample). Tell me: which column is your primary quality score, and what's the desired direction (higher-is-better / lower-is-better)? What's your production latency budget at p95? What's your monthly cost ceiling, and at roughly how many eval-equivalent runs/month? Which pricing.json version should I use?"

Step 2: Augment the Schema

Add cost_usd and confirm latency / token columns are populated. Flag any rows missing these - they get excluded from frontier analysis with an explicit count in the report.

Step 3: Per-Model Aggregates

For each model:

• Mean quality with 95% CI (bootstrap, 1000 resamples)
• p50, p95, p99 latency
• Mean cost-per-run and projected cost at the user's stated runs/month
• Failure rate (eval rows with no completion / timeout / parse error)

Step 4: Build the Pareto Frontier

A model is on the frontier iff no other model is strictly better on all three of (higher quality, lower cost, lower p95 latency). Output two tables:

• Quality vs Cost frontier
• Quality vs Latency frontier

Dominated models stay in the report but are flagged "dominated by X" with the reason.

Step 5: Quality-per-Dollar Composite

Q$ = quality_score / cost_per_1k_runs. Report rank order and a bootstrapped 95% CI on the rank - if two models' CIs overlap, the report says so explicitly instead of pretending the ranking is precise.

Step 6: Decision Matrix

A small table of "if your constraint is X, pick Y" recommendations - typically:

Step 7: Sensitivity Pass

Re-run the ranking with pricing perturbed ±20% and quality perturbed ±2 points. If the top recommendation flips under any plausible perturbation, the report flags it as "tight" and recommends collecting more eval samples before committing.

Demo Output Snippet

Frontier - Quality vs Cost (per 1k eval runs):

Decision (latency budget p95 < 2s, cost ceiling $10/1k runs):

Pick claude-sonnet-4-5. It's the highest-quality model that satisfies both constraints. opus-4-6 has +0.03 quality but breaks the latency budget; gpt-5 is dominated.

Tips

1. Record pricing_source on every row. Without it, "this model used to be a great deal" becomes unanswerable when prices move.
2. Don't average latency - use p95. A model with great median and a long tail is unsafe in production.
3. Treat the frontier as a working set, not a leaderboard. The "right" model depends on the deployment, not the eval.
4. Collect failures separately. A model that's 5% cheaper but 5% more likely to fail entirely is usually a bad trade - track failure rate as its own dimension.
5. Re-run on every model release. Frontiers shift quarterly; the model that won last quarter may be dominated this quarter.