Matscibench Eval

matscibench-eval

MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science — Zhang et al. (2025) (arXiv:2510.12171, 2025)

What this evaluates

This benchmark evaluates the reasoning capabilities of large language models in materials science, covering six primary fields and 31 sub-fields. It probes domain knowledge, mathematical/formula reasoning, and multimodal visual comprehension through expert-curated problems with three-tier difficulty classifications.

Datasets

• MatSciBench — total 1340; splits: test (-1); repo https://github.com/Jun-Kai-Zhang/MatSciBench.git

Metrics

• Accuracy Score (%) (primary) — range: percent
  • Percentage of correctly answered questions. Evaluated via a hybrid rule-based and LLM-based system. Rule-based checks apply a relaxed 5% numerical tolerance for calculations. For complex formulas/equations, Gemini-2.0-Flash provides the final correctness judgment.

Input / output format

Input: Text-based questions or multimodal questions containing images, presented with prompts for basic chain-of-thought (CoT), self-correction (3-round iterative refinement), or tool-augmentation (Python code generation and execution).

Output: Final answer string (numerical value, formula, or text). For tool-augmentation, the model outputs Python code which is executed to derive the answer.

Scoring recipe

def score_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if is_formula_type(gold):
            if llm_judge(pred, gold): correct += 1
        else:
            if rule_check(pred, gold, tolerance=0.05): correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Self-correction prompting often degrades performance across most models rather than improving it.
• Tool-augmentation yields inconsistent results; some models show significant accuracy drops when using code interpreters.
• Multimodal evaluation suffers from severe performance drops compared to text-only, largely due to visual context misinterpretation by models.

Evidence (verbatim from paper)

The performance in terms of accuracy score of all models on text-only questions is presented in Table 1, and the performance of multimodal models on images-included questions is presented in Table 2. The correctness of the output answers is evaluated using a hybrid approach that combines rule-based evaluation and LLM-based evaluation. We adapt the rule-based evaluation system from Qwen-2.5 Math (Yang et al., [2024a]). Following the previous works(Methani et al., [2020]; Gupta et al., [2024]), we apply a relaxed numerical tolerance of 5% to account for approximation errors in calculations and image recognition. To address the limitations of rule-based systems in handling complex formulas and equations, we supplement this approach with Gemini-2.0-Flash for formula-type questions. The LLM’s judgment serves as the final determinant of correctness for these complex mathematical expressions.

Citation

@misc{zhang2025matscibench,
  title={MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.12171}
}

• arXiv: 2510.12171