qhjqhj00 matvqa-eval
Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science — Wu et al. (2025) (arXiv:2505.18319, 2025)
What this evaluates
Evaluates multimodal large language models' ability to perform fine-grained visual-scientific reasoning in materials science. It probes structure-property-performance relationships through quantitative, comparative, causal, and hypothetical variation tasks, requiring models to integrate visual data from experimental figures with domain-specific knowledge rather than relying on textual shortcuts.
Datasets
- MatVQA — total 1325; splits: Causal (950), Quantitative (7), Comparative (-1), Hypothetical (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions. Calculated as the number of predictions matching the ground-truth answer divided by the total number of instances, reported per task split and overall.
Input / output format
Input: An experimental figure (e.g., material structure diagram or property plot) paired with a natural language question and multiple-choice options.
Output: A selected answer choice (letter or text) corresponding to the correct option, optionally accompanied by a chain-of-thought explanation.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
return (correct / len(gold)) * 100
Common pitfalls
- Models may exploit textual or caption shortcuts rather than performing genuine visual analysis, artificially inflating scores on raw questions.
- The Quantitative split contains only 7 items, so overall accuracy is heavily dominated by Causal and Comparative tasks and does not reliably reflect numeric reasoning capability.
- Domain-specific fine-tuning on unrelated visual modalities (e.g., optical chemical structures) can negatively bias performance on this benchmark.
Evidence (verbatim from paper)
The uniformly low accuracy proved that MatVQA is challenging for both large language models and small language models. These limitations likely stem from a combination of factors, including the nuanced visual perception required for material-scientific figures and the sophisticated reasoning demanded by tasks such as comparative and hypothetical analysis, which were identified as particularly challenging.
Citation
@misc{wu2025matvqa,
title={Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science},
author={Wu et al. (2025)},
year={2025},
note={arXiv:2505.18319}
}
- arXiv: 2505.18319