Megascience Eval

megascience-eval

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning — Fan et al. (2025) (arXiv:2507.16812, 2025)

What this evaluates

Evaluates large language models' scientific reasoning capabilities across general science, specialized domains (chemistry, CS, medicine, physics), and mathematical problem-solving. It tests the model's ability to follow chain-of-thought prompting, extract precise answers (including units), and correctly identify multiple-choice options.

Datasets

• MegaScience Evaluation Suite (MMLU, GPQA-Diamond, MMLU-Pro, SuperGPQA, SciBench, OlympicArena, ChemBench, CS-Bench, MedQA, MedMCQA, PubMedQA, PIQA, GSM8K, MATH, MATH500) — total ?; splits: test (-1)

Metrics

• EM (Exact Match) (primary) — range: [0, 1]
  • Exact match accuracy between the extracted model prediction and the gold answer. For computational problems requiring units, both the numerical value and the unit must match exactly.

Input / output format

Input: Benchmark questions (multiple-choice or computational problems) provided with chain-of-thought prompting templates.

Output: Free-text model responses containing reasoning steps and a final answer, which may be formatted using \boxed{}, option labels, or plain text.

Scoring recipe

def score(predictions, golds):
    matches = 0
    for pred, gold in zip(predictions, golds):
        # Stage 1: Identify answer indicator phrases
        answer = extract_by_indicator(pred)
        # Stage 2: Fallback to formatting patterns (\boxed{}, \mathrm{}, etc.)
        if not answer:
            answer = extract_by_format(pred)
        # Stage 3: For MCQs, match option content if label extraction fails
        if not answer:
            answer = match_option_label(pred)
        # Stage 4: Exact match (including unit if required)
        if normalize(answer) == normalize(gold):
            matches += 1
    return matches / len(predictions)

Common pitfalls

• Relying solely on \boxed{} extraction misses valid answers that lack this formatting, incorrectly penalizing models.
• Failing to match option labels for multiple-choice questions when direct extraction of the label fails.
• Ignoring unit requirements for computational problems, leading to false negatives when only the numerical value matches.

Evidence (verbatim from paper)

EM (unit) represents exact match accuracy for both the numerical answer and its corresponding unit. To enhance extraction precision, we develop a comprehensive set of rule-based methods tailored to extract answers across diverse question types. Our answer extraction method operates through a two-stage process: (1) identifying answer indicator phrases that signal the presence of a final answer, and (2) extracting the answer content from various formatting patterns.

Citation

@misc{fan2025megascience,
  title={MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning},
  author={Fan et al. (2025)},
  year={2025},
  note={arXiv:2507.16812}
}

• arXiv: 2507.16812