# Mattermech Eval

> Evaluates large language models' ability to reason about physicochemical principles in nanomaterial synthesis. It probes whether models can generate scientifically valid hypotheses and understand conceptual mechanisms from literature abstracts, rather than relying on abstract logic alone. Use when the user wants to benchmark on MatterMech, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/mattermech-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mattermech-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mattermech-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mattermech-eval

---


# mattermech-eval

> Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis — Pu et al. (2025) (arXiv:2509.25281, 2025)

## What this evaluates

Evaluates large language models' ability to reason about physicochemical principles in nanomaterial synthesis. It probes whether models can generate scientifically valid hypotheses and understand conceptual mechanisms from literature abstracts, rather than relying on abstract logic alone.

## Datasets

- **MatterMech** — total ?; splits: test (-1); repo https://github.com/amair-lab/MatterMech

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice (MCQ) and text-insertion format (TIF) questions. Calculated as the number of correct predictions divided by the total number of evaluated questions.

## Input / output format

**Input**: Context: paper abstract (and metadata). Prompt: A multiple-choice question or a text-insertion prompt requiring a single-word/phrase completion based on physicochemical principles.

**Output**: For MCQ: the selected option letter/text. For TIF: a single word or phrase.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Models may hallucinate physicochemical terminology not present in the source abstract, leading to biased or incorrect reasoning.
- Distractors in MCQs might be scientifically plausible or trivially incorrect, requiring careful validation to avoid easy guessing.
- TIF format evaluation is sensitive to exact string matching; minor formatting or synonym variations can incorrectly penalize valid answers.

## Evidence (verbatim from paper)

> Reviewer evaluates their accuracy and checks for potential hallucinations, and the Critic concludes the discussion upon reaching consensus.

## Citation

```bibtex
@misc{pu2025mattermech,
  title={Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis},
  author={Pu et al. (2025)},
  year={2025},
  note={arXiv:2509.25281}
}
```

- arXiv: 2509.25281