UitbreidenOS LLM Benchmarking Skill
When to activate
- Comparing two or more LLM models for a production use case
- Detecting quality regressions after prompt changes or model updates
- Establishing performance baselines before shipping an LLM feature
- Creating internal leaderboards for model selection
- Measuring latency, cost, and quality tradeoffs across model tiers
- Running canary tests before rolling out a new model version
When NOT to use
- One-off prompt experiments (use llm-eval instead)
- Training/fine-tuning model evaluation (different workflow)
- Simple unit testing of deterministic code paths
- When you need evaluation dataset design (use llm-eval)
Instructions
1. Benchmark Suite Design
Structure benchmarks by capability dimension:
benchmark_suite:
name: "customer-support-quality"
version: "1.2.0"
dimensions:
- name: accuracy
weight: 0.30
description: "Correctness of information provided"
metric: llm_as_judge
grader: gpt-4 # independent judge model
- name: helpfulness
weight: 0.25
description: "Actionability and completeness"
metric: llm_as_judge
grader: gpt-4
- name: safety
weight: 0.20
description: "No harmful, biased, or hallucinated content"
metric: rubric_check
rubric: safety-rubric-v2
- name: latency_p95
weight: 0.15
description: "Response time under load"
metric: timing
threshold_ms: 3000
- name: cost_per_query
weight: 0.10
description: "Token cost per query"
metric: token_cost
threshold_usd: 0.02
test_cases:
count: 200
distribution:
- simple_queries: 40%
- complex_reasoning: 30%
- edge_cases: 20%
- adversarial: 10%
2. Baseline Establishment
Before comparing models, establish a stable baseline:
class BenchmarkBaseline:
"""Store and compare against historical baselines."""
def establish(self, model: str, suite: BenchmarkSuite) -> Baseline:
"""Run benchmark 3x and average for stable baseline."""
runs = [suite.run(model) for _ in range(3)]
baseline = Baseline(
model=model,
suite=suite.name,
timestamp=datetime.now(),
scores={
dim.name: {
"mean": np.mean([r.scores[dim.name] for r in runs]),
"std": np.std([r.scores[dim.name] for r in runs]),
"p5": np.percentile([r.scores[dim.name] for r in runs], 5),
"p95": np.percentile([r.scores[dim.name] for r in runs], 95),
}
for dim in suite.dimensions
},
latency_p50=np.percentile(all_latencies, 50),
latency_p95=np.percentile(all_latencies, 95),
cost_per_query=np.mean(costs),
)
# Save as the canonical baseline
store.save(baseline, tag=f"baseline-{suite.name}-v{suite.version}")
return baseline
3. A/B Model Comparison
def compare_models(
baseline: Baseline,
challenger: str,
suite: BenchmarkSuite,
significance: float = 0.05,
) -> ComparisonReport:
"""Compare challenger against baseline with statistical rigor."""
# Run challenger multiple times for variance estimation
challenger_runs = [suite.run(challenger) for _ in range(5)]
report = ComparisonReport(baseline=baseline, challenger=challenger)
for dim in suite.dimensions:
baseline_scores = [baseline.scores[dim.name]["mean"]]
challenger_scores = [r.scores[dim.name] for r in challenger_runs]
# Statistical test (bootstrap for non-normal distributions)
t_stat, p_value = ttest_ind(baseline_scores, challenger_scores)
# Effect size (Cohen's d)
effect = cohens_d(challenger_scores, baseline_scores)
report.add_dimension(
name=dim.name,
baseline=baseline.scores[dim.name]["mean"],
challenger=np.mean(challenger_scores),
delta=np.mean(challenger_scores) - baseline.scores[dim.name]["mean"],
p_value=p_value,
significant=p_value < significance,
effect_size=effect,
verdict="WIN" if p_value < significance and effect > 0.2
else "LOSS" if p_value < significance and effect < -0.2
else "TIE",
)
return report
4. Regression Detection
def detect_regression(
current: BenchmarkResult,
baseline: Baseline,
thresholds: dict = None,
) -> list[Regression]:
"""Detect quality regressions against baseline."""
regressions = []
for dim in current.suite.dimensions:
current_score = current.scores[dim.name]
baseline_mean = baseline.scores[dim.name]["mean"]
baseline_std = baseline.scores[dim.name]["std"]
# Regression if score drops below baseline - 2σ
threshold = thresholds.get(dim.name, 2.0)
lower_bound = baseline_mean - threshold * baseline_std
if current_score < lower_bound:
regressions.append(Regression(
dimension=dim.name,
severity="CRITICAL" if current_score < baseline_mean - 3 * baseline_std
else "WARNING",
current=current_score,
baseline=baseline_mean,
drop=baseline_mean - current_score,
drop_pct=(baseline_mean - current_score) / baseline_mean * 100,
))
return regressions
5. Production Monitoring Dashboard
Track these metrics continuously:
┌─────────────────────────────────────────────────────┐
│ LLM BENCHMARK DASHBOARD │
├─────────────────────────────────────────────────────┤
│ │
│ QUALITY SCORES (daily) │
│ ┌──────────┬────────┬────────┬──────────┐ │
│ │ Metric │ Today │ 7-day │ Baseline │ │
│ ├──────────┼────────┼────────┼──────────┤ │
│ │ Accuracy │ 0.87 │ 0.85 │ 0.83 │ ✅ +4.8% │
│ │ Helpful │ 0.79 │ 0.81 │ 0.82 │ ⚠️ -1.2% │
│ │ Safety │ 0.98 │ 0.97 │ 0.96 │ ✅ +2.1% │
│ └──────────┴────────┴────────┴──────────┘ │
│ │
│ PERFORMANCE │
│ Latency P50: 1.2s | P95: 2.8s | P99: 4.1s │
│ Cost/query: $0.018 | Daily spend: $432 │
│ │
│ ALERTS │
│ ⚠️ Helpfulness trending down 3 days — investigate │
│ ✅ Accuracy improved after prompt v2.3 deploy │
│ │
└─────────────────────────────────────────────────────┘
6. Leaderboard Generation
MODEL LEADERBOARD: customer-support-quality v1.2
Benchmark date: 2026-06-13 | Test cases: 200
| Rank | Model | Overall | Accuracy | Helpful | Safety | Lat P95 | $/query |
|------|--------------------|---------|----------|---------|--------|---------|---------|
| 1 | claude-sonnet-4 | 0.91 | 0.93 | 0.89 | 0.99 | 2.1s | $0.015 |
| 2 | gpt-4o | 0.89 | 0.91 | 0.88 | 0.97 | 1.8s | $0.018 |
| 3 | claude-haiku-3.5 | 0.84 | 0.85 | 0.82 | 0.96 | 0.8s | $0.003 |
| 4 | gemini-2.5-flash | 0.82 | 0.84 | 0.79 | 0.95 | 1.1s | $0.004 |
| 5 | llama-3.1-405b | 0.78 | 0.80 | 0.76 | 0.92 | 3.2s | $0.008 |
Selection recommendation: claude-sonnet-4 (best overall, acceptable latency)
Budget option: claude-haiku-3.5 (92% of top quality at 20% cost)
Example
Comparing model upgrade for production chatbot:
BENCHMARK REPORT: Customer Support Bot v2.3 → v2.4
Suite: customer-support-quality v1.2 | 200 test cases | 5 runs
BASELINE: claude-sonnet-4 + prompt v2.3
CHALLENGER: claude-sonnet-4 + prompt v2.4
DIMENSION COMPARISON:
| Dimension | Baseline | Challenger | Delta | p-value | Effect | Verdict |
|------------|----------|------------|--------|---------|--------|---------|
| Accuracy | 0.89 | 0.93 | +0.04 | 0.003 | 0.67 | ✅ WIN |
| Helpfulness| 0.82 | 0.81 | -0.01 | 0.42 | 0.12 | TIE |
| Safety | 0.97 | 0.98 | +0.01 | 0.31 | 0.15 | TIE |
| Latency P95| 2.3s | 2.1s | -0.2s | 0.01 | 0.45 | ✅ WIN |
| Cost/query | $0.019 | $0.015 | -$0.004| <0.001 | 0.89 | ✅ WIN |
RECOMMENDATION: Ship v2.4 — accuracy improved 4.5% with faster latency and lower cost.
No regression detected. Helpfulness unchanged (TIE, not regression).
Anti-Patterns
- Single-run comparison: Always run 3-5 times minimum — LLM outputs are stochastic
- Ignoring variance: A "win" within noise bounds is not a real win
- Benchmark gaming: Optimizing for benchmark metrics at the expense of real-world quality
- Stale baselines: Re-establish baselines quarterly as model landscape evolves
- Cost-blind comparison: The best model is not always the most accurate — consider cost/quality tradeoff
- No adversarial tests: Only testing happy paths misses failure modes