qhjqhj00 climabench-eval
ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis — Wang et al. (2026) (arXiv:2604.16922, 2026)
What this evaluates
This benchmark evaluates LLM-based agents on autonomous, open-ended climate science problem-solving. It probes the model's ability to perform data-driven modeling, apply physics-aware constraints, and generate scientifically rigorous analysis reports without human intervention.
Datasets
- ClimaBench — total 320; splits: test (320); repo https://github.com/usail-hkust/ClimAgent
Metrics
Overall(primary) — range: [0, 10]- Multi-dimensional evaluation rubric assessing solution quality across AE, SC, PS, RBA, and MR. Scores are averaged into an Overall metric.
AE— range: [0, 10]- Accuracy/Effectiveness score assessing task comprehension and solution correctness.
MR— range: [0, 10]- Modeling Rigor score evaluating the scientific soundness and constraint reasoning of the generated model.
PS— range: [0, 10]- Problem Solving score measuring the agent's ability to construct coherent, solution-aware models.
RBA— range: [0, 10]- Robustness/Breadth score evaluating performance across diverse problem types and domains.
SC— range: [0, 10]- Scientific Correctness score verifying the rationality and expert-level validity of the solution.
Input / output format
Input: Open-ended climate science problems curated from research papers published between 2000 and 2025.
Output: Autonomous scientific models, analysis reports, or solutions generated by the LLM agent.
Scoring recipe
def evaluate_climabench(solutions, problems):
scores = {}
for sol, prob in zip(solutions, problems):
# Automatic LLM scoring via GPT-4o
llm_score = gpt4o.evaluate(sol, prob, rubric=["AE", "SC", "PS", "RBA", "MR"])
scores[prob.id] = llm_score
# Human expert verification on 100 random samples
human_scores = [expert.evaluate(sol, prob) for sol, prob in random.sample(zip(solutions, problems), 100)]
# Aggregate dimensions and compute Overall
overall = mean([mean(dim_scores) for dim_scores in scores.values()])
return overall
Common pitfalls
- Data leakage: Problems from 2025 are evaluated separately from 2000–2024 to prevent memorization of pretraining data.
- Subjectivity in scoring: Automatic LLM scoring requires human expert verification to ensure scientific validity and prevent hallucination bias.
Evidence (verbatim from paper)
ClimAgent consistently outperforms all ablated variants under five evaluation metrics. Removing PA significantly reduces AE and MR, indicating that deep task comprehension is essential for rigorous formulation. The absence of CM leads to sharp declines in MR and PS, highlighting its critical role in constructing coherent, scientifically sound models.
Citation
@misc{wang2026climagent,
title={ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis},
author={Wang et al. (2026)},
year={2026},
note={arXiv:2604.16922}
}
- arXiv: 2604.16922