Climategpt Eval

climategpt-eval

ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change — Thulke et al. (2024) (arXiv:2401.09646, 2024)

What this evaluates

Evaluates large language models on climate-specific knowledge, reasoning, and fact-verification, alongside general domain benchmarks for commonsense reasoning and world knowledge. It also tests multilingual capability via cascaded machine translation on an Arabic exam dataset.

Datasets

• ClimaBench — total ?; splits: test (-1)
• Pira 2.0 MCQ — total ?; splits: test (-1)
• Exeter Misinformation — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• PIQA — total ?; splits: test (-1)
• OpenBookQA — total ?; splits: test (-1)
• WinoGrande — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• EXAMS (Arabic) — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: percent
  • Accuracy is computed as the fraction of correctly predicted labels or answers. For ClimaBench, a dataset-weighted average of individual task accuracies is reported.

Input / output format

Input: Text classification prompts or multiple-choice questions with concatenated choices. For fact-verification, claim-evidence pairs or claims alone.

Output: Predicted class label or selected multiple-choice option.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) * 100

Common pitfalls

• Models are evaluated using log-probability ranking for MCQs rather than generation, which can favor models with better calibration over actual reasoning.
• ClimaBench results are reported as a weighted average, masking per-dataset performance variations.
• Cascaded MT evaluation introduces translation errors that can unfairly penalize models not natively trained in the target language.

Evidence (verbatim from paper)

Our primary evaluation format involves text classification and multiple-choice questions (MCQs), structured as log probability ranking tasks. Table 11 shows the 5-shot results of all models on the set of climate-specific benchmarks. We report 10-shot results on HellaSwag and 5-shot on all other benchmarks. EXAMS (Acc [%])

Citation

@misc{thulke2024climategpt,
  title={ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change},
  author={Thulke et al. (2024)},
  year={2024},
  note={arXiv:2401.09646}
}

• arXiv: 2401.09646