Msqa Eval

msqa-eval

MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge — Cheung et al. (2025) (arXiv:2505.23982, 2025)

What this evaluates

Evaluates graduate-level materials science reasoning and factual knowledge through long-form explanatory answers and binary true/false questions. It probes model capabilities in domain-specific knowledge retrieval, multi-step scientific reasoning, and accuracy under both direct prompting and retrieval-augmented generation (RAG) settings.

Datasets

• MSQA — total 1757; splits: test (1757); repo https://github.com/jerry3027/MSQA

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For binary tasks, exact keyword matching for 'YES' or 'NO'. For long-answer tasks, GPT-4o judge classifies responses as 'correct', 'mostly correct', or 'incorrect'; the first two are counted as correct.

Input / output format

Input: Question text. Optionally augmented with top-5 BM25-retrieved paragraphs from Methods/Results sections. For binary tasks, optionally includes chain-of-thought prompting instructions.

Output: Long-answer: free-text scientific explanation. Binary-answer: exact string 'YES' or 'NO'.

Scoring recipe

def compute_accuracy(preds, golds, task_type):
    correct = 0
    for p, g in zip(preds, golds):
        if task_type == 'binary':
            if p.strip().upper() in ('YES', 'NO') and p.strip().upper() == g.strip().upper():
                correct += 1
        elif task_type == 'long':
            judge_label = llm_judge_eval(p, g)
            if judge_label in ('correct', 'mostly correct'):
                correct += 1
    return (correct / len(preds)) * 100

Common pitfalls

• Domain-specific fine-tuned models often underperform general-purpose LLMs due to overfitting and distributional shifts between their training data and the benchmark.
• Chain-of-thought prompting can degrade binary-answer accuracy by introducing factually incorrect intermediate reasoning steps.
• LLM-as-judge (e.g., GPT-4o-mini) exhibits verbosity bias and may accept vague explanations, requiring validation against stronger judges or human experts.

Evidence (verbatim from paper)

For binary-answer evaluations, accuracy is determined by exact keyword matching for responses containing either “YES” or “NO”. Performance results are uniformly reported as accuracy percentages across all experiments. Evaluating the long-answer responses is conducted through GPT-4o acting as an LLM judge, assessing responses as either “correct” or “mostly correct”, both categories counted as correct in our metrics.

Citation

@misc{cheung2025msqa,
  title={MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge},
  author={Cheung et al. (2025)},
  year={2025},
  note={arXiv:2505.23982}
}

• arXiv: 2505.23982