qhjqhj00 mascqa-eval
Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering — Bajan et al. (2025) (arXiv:2501.04277, 2025)
What this evaluates
Evaluates large language models' domain-specific reasoning and numerical problem-solving capabilities in materials science and metallurgical engineering. It probes their ability to accurately answer multiple-choice, matching, and numerical questions, highlighting gaps in scientific reasoning and computational precision.
Datasets
- MaScQA — total 644; splits: test (644); repo https://github.com/Lambard-ML-Team/LLM
Metrics
accuracy(primary) — range: percent- Percentage of questions answered correctly, averaged over three submissions per question. Correctness is determined by an AI judge that verifies answer equivalence and reasoning soundness.
Input / output format
Input: A prompt containing the question, the correct answer, the model's predicted answer, and the question type (MCQ, MATCH, MCQN, or NUM).
Output: A structured judgment block: \nExplanation: [brief explanation]\nResult: [CORRECT or INCORRECT]\n
Scoring recipe
correct_count = 0
total = len(dataset) * 3
for q in dataset:
for _ in range(3):
pred = model.generate(q)
judge_output = run_judge(q, q.correct_answer, pred, q.type)
if 'CORRECT' in judge_output:
correct_count += 1
accuracy = (correct_count / total) * 100
Common pitfalls
- Numerical questions (NUM/MCQN) are particularly challenging due to tokenization inefficiencies and multi-step reasoning requirements, often causing models to perform near random baseline.
- Models may select correct options in MCQ/MATCH categories through pattern exploitation or guessing without sound underlying reasoning.
- Hardware constraints (e.g., running on MAC vs GPU) significantly impact open-source model performance and stability.
Evidence (verbatim from paper)
a) Based on the question and the correct answer, You must tell if the other answer is correct or not by answering only with Correct or Incorrect ... The evaluation of the LLMs, on Table 5 and Figure 6, demonstrates that Claude-3.5-Sonnet and GPT-4o are among the top performers, achieving overall accuracies of approximately 84% (see Figure 1 in the supplementary materials for details concerning the LLMs' average accuracy on each category MATCH, MCQ, MCQN, and NUM).
Citation
@misc{bajan2025exploring,
title={Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering},
author={Bajan et al. (2025)},
year={2025},
note={arXiv:2501.04277}
}
- arXiv: 2501.04277