Climate Eval Eval

climate-eval-eval

Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change — Kurfalı et al. (2025) (arXiv:2505.18653, 2025)

What this evaluates

This benchmark evaluates open-source large language models on their ability to understand, classify, and reason about climate-related discourse. It probes capabilities across text classification, stance detection, claim verification, misinformation detection, and named entity recognition using real-world news, corporate reports, social media, and scientific abstracts.

Datasets

• Guardian Climate News Corpus — total 40173; splits: test (-1); HF NLP-RISE/guardian_climate_news_corpus; repo https://github.com/NLP-RISE/extractguardian
• Climate-Stance — total 3777; splits: test (-1)
• Climate-FEVER — total 7675; splits: test (-1)
• Climate-Change NER — total 534; splits: test (-1)
• Net-Zero Reduction — total 3517; splits: test (-1)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • The unweighted mean of recall (or precision) computed for each class independently, then averaged across all classes. Handles class imbalance by treating all classes equally.

Input / output format

Input: Text input varying by task: article titles, article bodies, or both; claim-evidence pairs; multiple-choice questions with or without context; or raw text passages for stance/sentiment/entity classification.

Output: Discrete class label (e.g., topic, stance, sentiment), selected multiple-choice option, or token-level entity spans. Models are prompted in zero-shot or 5-shot settings.

Scoring recipe

def compute_macro_f1(predictions, gold_labels, classes):
    per_class_f1 = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        per_class_f1.append(f1)
    return sum(per_class_f1) / len(classes)

Common pitfalls

• Task variants exist for the same dataset (e.g., Guardian Title vs Body, SciDCC Title/Summary/Body); evaluators must match the exact variant specified.
• Climate-FEVER is evaluated as a three-way entailment task per claim-evidence pair (7,675 instances), not using the original aggregated general label.
• PIRA tasks use accuracy instead of macro-F1, as noted in the table caption.

Evidence (verbatim from paper)

The numbers indicate the models’ performance for each task in F1-macro, except for PIRA (see Section 3.2).

Citation

@misc{kurfali2025climateeval,
  title={Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change},
  author={Kurfalı et al. (2025)},
  year={2025},
  note={arXiv:2505.18653}
}

• arXiv: 2505.18653