Clinb Eval

clinb-eval

CLINB: A Climate Intelligence Benchmark for Foundational Models — Huebscher et al. (2025) (arXiv:2511.11597, 2025)

What this evaluates

Evaluates foundational models on climate intelligence by testing their ability to generate long-form, evidence-grounded answers with accurate citations and relevant multimodal content. It probes knowledge synthesis, hallucination rates in references and images, and alignment with expert-curated quality rubrics.

Datasets

• CLINB — total ?; splits: test (-1)

Metrics

• ELO score (primary) — range: other
  • Computed using the Bradley-Terry model on pairwise battle outcomes (winner/tie). Includes bootstrap 95% confidence intervals. Evaluated across overall preference and sub-dimensions (Citations, Images, Knowledge, Presentation).

Input / output format

Input: Question, system prompt specifying expected answer format, quality dimensions, role of references, and instructions to include URLs for evidence and images.

Output: Long-form answer text containing factual claims, inline references with URLs, and embedded images (or links to them).

Scoring recipe

battles = generate_all_pairs(answers)
for q, ans_a, ans_b in battles:
    # Swap order to control position bias
    judge_input = format_pair(q, ans_a, ans_b)
    judge_output = judge_model(judge_input)  # Gemini 2.5 Pro
    winner = select_winner(judge_output)     # a, b, or tie
    record_battle(q, winner)
elo_scores = bradley_terry_model(battles, bootstrap_ci=0.95)

Common pitfalls

• Position bias in pairwise comparisons (controlled by swapping answer order).
• Judge model anchoring bias: may overvalue strong overall outputs and underpenalize missing images.
• Human expert familiarity bias: may prefer presentation styles seen during data creation over factual rigor.
• Rubric incompleteness: autorater may reward ungrounded claims that match rubric keywords if specific rubrics are used.

Evidence (verbatim from paper)

We perform model assessment of answer pairs: two answers are evaluated side by side (SxS) to identify the preferred one. We adapt the Chatbot Arena’s battles setup (Chiang et al., [2024b]). A single battle involves one question and two systems answers. We run three rounds of evaluations. In each round, we execute two pairwise evaluations, swapping the order of the answers to control for position bias (Wang et al., [2023]). The system with the majority score is the winner; otherwise, it is a tie. This procedure defines a single battle. We compute battles between all pairs of sources answers, 4147 battles total. We estimate the ELO scores for all systems using the Bradley-Terry model, including a bootstrap 95% confidence interval.

Citation

@misc{huebscher2025clinb,
  title={CLINB: A Climate Intelligence Benchmark for Foundational Models},
  author={Huebscher et al. (2025)},
  year={2025},
  note={arXiv:2511.11597}
}

• arXiv: 2511.11597