Macbench Eval

macbench-eval

Probing the limitations of multimodal language models for chemistry and materials research — Alampara et al. (2024) (arXiv:2411.16955, 2024)

What this evaluates

Evaluates vision-language models' ability to perform multimodal scientific reasoning in chemistry and materials research. It probes capabilities across data extraction, experimental understanding, and data interpretation, specifically testing spatial reasoning, cross-modal synthesis, and multi-step inference.

Datasets

• MaCBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly answered questions out of the total number of instances. The paper also reports performance normalized as the fraction of correctly answered questions relative to a random baseline.

Input / output format

Input: Multimodal inputs including scientific images (e.g., crystal structures, AFM images, plots, tables, chemical diagrams), text descriptions, and spectral data, paired with domain-specific questions or prompts.

Output: Textual answers to questions, typically multiple-choice selections or direct responses to extraction/interpretation queries.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = correct / len(gold_labels)

Common pitfalls

• Models excel at superficial perception tasks but fail fundamentally on spatial reasoning and multi-step scientific inference.
• Performance is highly sensitive to prompt wording and scientific terminology; minor changes (e.g., 'image' vs 'diagram') cause large accuracy swings.
• High correlation with web search frequency suggests models often rely on training data pattern-matching rather than genuine reasoning.

Evidence (verbatim from paper)

While the best models perform well at extracting information about reaction diagrams, they fail to correctly describe the relationship between isomers (see Figure A.3). As discussed below, this is likely caused by models struggling with spatial reasoning. In addition, even the extraction of compositions from tables still shows room for improvement for the VLLMs we tested (average accuracy of 0.53), performing not distinguishable from random guessing for Llama 3.2 90B Vision.

Citation

@misc{alampara2024macbench,
  title={Probing the limitations of multimodal language models for chemistry and materials research},
  author={Alampara et al. (2024)},
  year={2024},
  note={arXiv:2411.16955}
}

• arXiv: 2411.16955