Mind Benchmark Eval

mind-benchmark-eval

MIND: AI Co-Scientist for Material Research — Ahn et al. (2026) (arXiv:2604.13699, 2026)

What this evaluates

Evaluates an AI co-scientist framework's ability to automatically validate materials science hypotheses using MLIP-based simulations. It measures both binary verification accuracy across energetic, mechanical, and structural categories, and human-rated scientific utility via expert feedback.

Datasets

• MIND MLIP-expert-curated benchmark — total 28; splits: test (28); repo https://github.com/IMMS-Ewha/MIND

Metrics

• accuracy (primary) — range: [0, 1]
  • Binary yes/no verification accuracy. Calculated as the number of correctly validated hypotheses divided by the total number of hypotheses (21/28 = 75.0%).
• user_rating — range: [1, 7]
  • Average score on a 7-point Likert scale across three dimensions: scientific validity, reasoning transparency, and research usefulness.

Input / output format

Input: A materials science hypothesis/claim requiring simulation-verifiable ground truth, categorized as energetic, mechanical, or structural.

Output: A binary verification result (yes/no) along with a reasoning trace and predicted validation result.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_likert_scores(user_ratings):
    return {dim: sum(scores)/len(scores) for dim, scores in user_ratings.items()}

Common pitfalls

• The benchmark is very small (28 hypotheses), so accuracy scores may not generalize to broader or out-of-distribution materials science domains.
• The reported speedup (36–72×) is relative to a specific human workflow using SevenNet-omni and may not hold for different simulation tools or human expertise levels.
• User study scores are subjective, rely on a single 7-point Likert scale, and do not mention blind evaluation or statistical significance testing.

Evidence (verbatim from paper)

Across 28 hypothesis, MIND correctly validates 21 cases, achieving an overall accuracy of 75.0%, with category accuracies of $70%$ (energetic), $75%$ (structural), and $100%$ (mechanical). On average, MIND verifies a hypothesis in 5 minutes. Compared to the typical 3–6 hour human research loop using SevenNet-omni, this represents a 36–72$\times$ speedup in hypothesis verification.

Citation

@misc{ahn2026mind,
  title={MIND: AI Co-Scientist for Material Research},
  author={Ahn et al. (2026)},
  year={2026},
  note={arXiv:2604.13699}
}

• arXiv: 2604.13699