Climaqa Eval

climaqa-eval

ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models — Manivannan et al. (2024) (arXiv:2410.16701, 2024)

What this evaluates

Evaluates LLMs on climate science question-answering across multiple formats (multiple-choice, freeform, cloze) and complexity levels (base, reasoning, hypothetical). It probes factual recall, scientific reasoning, and the impact of adaptation techniques like RAG, few-shot prompting, and fine-tuning.

Datasets

• ClimaQA — total 3566; splits: test (3566); repo https://github.com/Rose-STL-Lab/genie-climaqa

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions where the model's top predicted token matches the ground-truth option letter.
• exact-match (EM) — range: percent
  • Exact string match between the model's generated single word and the ground-truth answer for cloze questions.
• partial-score (PS) — range: [0, 1]
  • Token-level or word-level overlap score for cloze questions, ranging from 0 to 1.
• BLEU — range: percent
  • Standard n-gram overlap metric for freeform answers.
• BERTScore — range: [0, 1]
  • Contextual embedding-based similarity metric for freeform answers.
• factual_accuracy — range: [0, 1]
  • Expert or automated metric designed to measure scientific correctness without bias towards the QA-generation model.

Input / output format

Input: Question text, optionally with multiple-choice options or a blank for cloze. For RAG settings, relevant textbook chunks are appended as context.

Output: Single uppercase letter (MCQ), single scientific word (Cloze), or a concise answer of maximum 2 sentences (Freeform).

Scoring recipe

def score_mcq(pred, gold):
    return 1.0 if pred.strip().upper() == gold.strip().upper() else 0.0
def score_cloze_em(pred, gold):
    return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0
def score_cloze_ps(pred, gold):
    return len(set(pred.lower().split()) & set(gold.lower().split())) / max(len(gold.lower().split()), 1)
def aggregate_accuracy(predictions, golds):
    return sum(score_mcq(p, g) for p, g in zip(predictions, golds)) / len(golds) * 100

Common pitfalls

• BLEU and BERTScore scores are biased towards GPT-3.5-turbo, the model used to generate the synthetic questions, making them less reliable for cross-model comparison.
• RAG performance heavily depends on retrieval source: retrieving from textbooks used to generate questions (source) boosts scores, while retrieving from held-out textbooks often degrades performance due to distracting content.
• Freeform questions often yield higher reasoning scores than MCQs for the same model, indicating that forced-choice formats may constrain complex scientific reasoning.

Evidence (verbatim from paper)

Finally, We observe that the BLEU and BERTScore metrics are slightly biased towards the model that was used for QA-generation (gpt-3.5-turbo) while this is not seen in the proposed Factual Accuracy metric [5].

Citation

@misc{manivannan2024climaqa,
  title={ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models},
  author={Manivannan et al. (2024)},
  year={2024},
  note={arXiv:2410.16701}
}

• arXiv: 2410.16701