Enviroexam Eval

enviroexam-eval

EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models — Huang et al. (2024) (arXiv:2405.11265, 2024)

What this evaluates

This benchmark evaluates large language models' domain-specific knowledge in environmental science using multiple-choice questions derived from university curricula. It measures both raw accuracy and performance consistency across different course topics, revealing how well models retain and apply specialized scientific concepts.

Datasets

• EnviroExam — total 936; splits: dev (210), test (726)

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly answered multiple-choice questions per model across all test items.
• composite_index (primary) — range: [0, 1]
  • I = M × (1 − CV), where M is the mean accuracy across tests and CV is the coefficient of variation (σ/M). If CV > 1, the score is 'model void'.

Input / output format

Input: Multiple-choice questions covering 42 environmental science courses, evaluated in 0-shot and 5-shot prompt settings.

Output: A single selected answer choice per question.

Scoring recipe

accuracies = [1 if pred == gold else 0 for pred, gold in zip(predictions, gold_labels)]
M = sum(accuracies) / len(accuracies)
sigma = (sum((a - M)**2 for a in accuracies) / len(accuracies)) ** 0.5
CV = sigma / M if M > 0 else float('inf')
if CV <= 1:
    composite_index = M * (1 - CV)
else:
    composite_index = 'model void'

Common pitfalls

• The composite index penalizes high variance in performance across sub-topics, so a model with high average accuracy but poor consistency on specific courses will score lower.
• If CV exceeds 1, the model is marked 'void' rather than receiving a negative score, which can disproportionately affect aggregate rankings if not explicitly handled.
• Evaluations are run in both 0-shot and 5-shot settings; results must be reported separately to avoid conflating zero-shot knowledge with few-shot prompting effects.

Evidence (verbatim from paper)

EnviroExam uses accuracy as the basis for scoring each subject’s questions and employs a comprehensive metric when calculating the total score. ... Calculate the coefficient of variation (CV): The coefficient of variation is the ratio of the standard deviation to the mean and is used to measure the relative dispersion of the scores: ... Calculate the original composite index I: I = M × (1 − CV), 0 ≤ CV ≤ 1; model void, CV > 1

Citation

@misc{huang2024enviroexam,
  title={EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2405.11265}
}

• arXiv: 2405.11265