Model Comparison

Model Comparison

Core Rule

Use published evidence before judgment. A model comparison can include analysis, but benchmark values must be traceable to a source, date, model version, benchmark version, harness details, and caveats.

If a model is unreleased, gated, internal, preview-only, or rumored, label it directly and do not invent scores. Compare it through disclosed signals: official statements, system cards, safety cards, partner reports, benchmark deltas from related public models, and availability/pricing language.

Workflow

1. Define the decision: coding agent, general assistant, research, enterprise workflow, cost routing, long context, multimodal, safety, or consumer use.
2. Build a source set with this priority: official system cards and model docs, benchmark-owner leaderboards/papers, official release posts, reputable independent evaluators, then high-quality secondary analysis. Use forum posts only as anecdotal field reports.
3. Capture exact model identifiers and dates. Include aliases only after the official name, such as claude-opus-4-8.
4. Capture pricing and total task-cost data before judging benchmark value. Token price alone is not enough; include input tokens, cached input tokens, output tokens, reasoning tokens, tool calls, retries, batch discounts, context-length surcharges, storage or file-search fees, and the benchmark's average token burn when available.
5. Separate benchmark tables into three groups:
   • Coding and agentic engineering: SWE-bench Verified/Pro/Multilingual/Multimodal, Terminal-Bench, FrontierSWE, ProgramBench, LiveCodeBench, SciCode, OSWorld, WebArena, MCP Atlas, Toolathlon, AutomationBench, tau-bench, BrowseComp when used by agents.
   • Tasks and life: GPQA, Humanity's Last Exam, MMLU/GMMLU, MMMU, MathArena/USAMO/AIME when not saturated, OfficeQA, Finance Agent, GDPval, HealthBench, Legal Agent Benchmark, Vending-Bench, ChartQA/ChartQAPro, ScreenSpot, multilingual and accessibility benchmarks.
   • Design and artifact creation: Design Arena or WebDev Arena for human preference over generated interfaces; Design2Code or FigmaBench for screenshot/design-to-code fidelity; Vibe Code Bench for functional zero-to-one web apps; PPTBench or SlidesBench for presentation layout and generation.
6. Normalize each score with benchmark version, metric, direction, harness, tools/no-tools status, effort/reasoning setting, context budget, trials, average total tokens per attempt, and whether the source warns about contamination or non-comparability. For design, also capture the evaluation dimension, reference/no-reference setup, viewport/device suite, framework constraint, asset and image access, render/runtime success, human vote count and confidence interval, judge model, and output editability.
7. Prefer workload conclusions over universal winners. Say which model deserves first evaluation for each workflow, which model wins on raw score, and which wins on cost-normalized score.
8. Add a verification note for any claim that changed recently: publication date, source URL, and whether the model is generally available, preview, gated, or internal.

Benchmark Selection

Read references/benchmark-taxonomy.md when choosing which benchmarks to include or when deciding whether a score is comparable.

Use a broad table only when it helps the reader. If the article is about coding, lead with engineering and tool-use benchmarks. If it is for executives or everyday buyers, lead with professional-work, search, document, safety, cost, and availability signals.

For design comparisons, match the benchmark to the intended work. Do not collapse aesthetics, reference fidelity, functionality, responsiveness, accessibility, and editability into one score. Lead with the matching benchmark family and show the other dimensions separately.

Required Output Tables

Include a concise top-model summary table in every substantive comparison. Show the exact model and effort, principal benchmark score, API input/output price, and measured task cost when available.

For every broad frontier-model comparison or software-related request, also include a dedicated software-engineering agent table in addition to other benchmark tables. Use the latest comparable agent-harness evaluation and include:

• Name the agent harness, such as OpenAI Codex or Claude Code; do not present harness-dependent results as pure model scores.
• Prefer current end-to-end components such as DeepSWE, Terminal-Bench, and SWE-Atlas-QnA. Substitute the current suite only when the benchmark owner changes it.
• Include turns/task and cache-hit rate when published and decision-relevant.
• Sort primarily by aggregate coding score, then by task cost.
• Keep different benchmark or index versions in separate columns or tables. Never silently mix them.
• Use - plus "no comparable public score" when evidence is missing.
• Do not substitute a SWE-bench-only table for this agent table. Raw model coding benchmarks may appear as an additional table.

For every design, frontend, presentation, or artifact-generation comparison, also include:

• Keep Design Arena or WebDev Arena categories separate, including agentic versus non-agentic tracks.
• Label human preference, screenshot/reference fidelity, functional workflow success, and presentation layout as different dimensions; never rank them as interchangeable.
• For preference arenas, capture vote count and confidence interval when published.
• For design-to-code, capture input type, viewport, framework, render success, metric family, and human evaluation.
• For generated products, capture browser workflows, allowed tools, asset/image access, and platform constraints.
• For presentations, capture template access, editability, overflow/overlap/readability checks, and whether scoring uses humans, deterministic checks, or an LLM judge.
• Treat Vibe Code Bench as a functional product-implementation benchmark rather than a visual-design benchmark.
• Use - for missing cost, time, or score data; do not invent conversions.

Evidence Matrix

Use scripts/benchmark_table.py to validate and render a JSON evidence file when a comparison has many records.

Minimum record fields:

{
  "benchmark": "SWE-bench Pro",
  "category": "coding-agent",
  "model": "Claude Opus 4.8",
  "score": "69.2%",
  "metric": "pass rate",
  "source_title": "Claude Opus 4.8 System Card",
  "source_url": "https://www.anthropic.com/claude-opus-4-8-system-card",
  "source_type": "system_card",
  "date": "2026-05-28",
  "notes": "Adaptive thinking at max effort; averaged over 5 trials."
}

Optional cost fields:

{
  "input_price_per_million": "5.00",
  "cached_input_price_per_million": "0.50",
  "output_price_per_million": "25.00",
  "sample_input_tokens": "100000",
  "sample_cached_input_tokens": "0",
  "sample_output_tokens": "30000",
  "sample_reasoning_tokens": "0",
  "tool_cost_usd": "0.00",
  "total_task_cost_usd": "1.25",
  "cost_basis": "Example 100k input + 30k output coding-agent task, no cache, no tool fees."
}

Run:

python3 scripts/benchmark_table.py evidence.json

Writing Standard

State confidence plainly:

• "Published benchmark leader" when the source gives a comparable published score.
• "Best first evaluation candidate" when scores are close or harnesses differ.
• "Not directly comparable" when vendors used different harnesses, tool access, budgets, or benchmark versions.
• "Unpublished" or "gated preview" when broad users cannot run the model.
• "No public score found" instead of filling the gap with rumor.

Cost language must be explicit:

• "Raw benchmark leader" for the highest published score.
• "Cost-normalized leader" for the best score per dollar under a stated token mix.
• "Cheapest token route" only when judging price alone, without any quality claim.
• "Unknown total cost" when a model has private routing, unpublished reasoning-token usage, tool fees, context surcharges, or subscription limits that prevent a clean task-cost estimate.

End with an evaluation plan: the top two or three models to test, the user's real tasks, expected failure modes, expected total token and tool-call cost, and the minimum local benchmark needed before migration.