Materialfigbench Eval

materialfigbench-eval

MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models — Yoshitake et al. (2026) (arXiv:2603.11414, 2026)

What this evaluates

Evaluates multimodal large language models' ability to solve college-level materials science problems that require accurate visual interpretation of scientific figures, such as phase diagrams and stress-strain curves, alongside domain-specific textual reasoning.

Datasets

• MaterialFigBENCH — total 137; splits: test (137)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the number of correct responses divided by the number of sampled responses per problem, then averaged across all 137 problems. A response is considered correct if it falls within the expert-defined answer range.

Input / output format

Input: A textual problem statement accompanied by a scientific figure (e.g., phase diagram, stress-strain curve, Arrhenius plot) that is essential for solving the problem.

Output: Free-response numerical or textual answer. Models must provide values with appropriate significant digits to match expert-defined answer ranges.

Scoring recipe

total_correct = 0
total_samples = 0
for problem in dataset:
    samples = model.generate(problem.text, problem.figure)
    for sample in samples:
        total_samples += 1
        if is_within_expert_range(sample.answer, problem.answer_range):
            total_correct += 1
accuracy = total_correct / total_samples

Common pitfalls

• Models often ignore the provided figures and rely on memorized domain knowledge (e.g., assuming a hypothetical element is carbon and using memorized solubility limits).
• Significant digit handling critically affects correctness; answers outside the expert-defined range due to rounding or sig-fig differences are marked incorrect.
• API and ChatGPT interfaces for the same model version can yield different accuracy scores due to sampling differences and interface-specific behaviors.

Evidence (verbatim from paper)

To obtain an accuracy value, we divided the number of correct responses by the number of sampled responses (10 for GPT-4o and 8 for GPT-o1 and GPT-5) and then averaged over all 137 problems.

Citation

@misc{yoshitake2026materialfigbench,
  title={MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models},
  author={Yoshitake et al. (2026)},
  year={2026},
  note={arXiv:2603.11414}
}

• arXiv: 2603.11414