Mattermech Eval

mattermech-eval

Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis — Pu et al. (2025) (arXiv:2509.25281, 2025)

What this evaluates

Evaluates large language models' ability to reason about physicochemical principles in nanomaterial synthesis. It probes whether models can generate scientifically valid hypotheses and understand conceptual mechanisms from literature abstracts, rather than relying on abstract logic alone.

Datasets

• MatterMech — total ?; splits: test (-1); repo https://github.com/amair-lab/MatterMech

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered multiple-choice (MCQ) and text-insertion format (TIF) questions. Calculated as the number of correct predictions divided by the total number of evaluated questions.

Input / output format

Input: Context: paper abstract (and metadata). Prompt: A multiple-choice question or a text-insertion prompt requiring a single-word/phrase completion based on physicochemical principles.

Output: For MCQ: the selected option letter/text. For TIF: a single word or phrase.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(predictions)

Common pitfalls

• Models may hallucinate physicochemical terminology not present in the source abstract, leading to biased or incorrect reasoning.
• Distractors in MCQs might be scientifically plausible or trivially incorrect, requiring careful validation to avoid easy guessing.
• TIF format evaluation is sensitive to exact string matching; minor formatting or synonym variations can incorrectly penalize valid answers.

Evidence (verbatim from paper)

Reviewer evaluates their accuracy and checks for potential hallucinations, and the Critic concludes the discussion upon reaching consensus.

Citation

@misc{pu2025mattermech,
  title={Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis},
  author={Pu et al. (2025)},
  year={2025},
  note={arXiv:2509.25281}
}

• arXiv: 2509.25281