Crystal Structure Discovery Eval

crystal-structure-discovery-eval

Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)

What this evaluates

Evaluates an LLM's capacity to discover stable crystal structures by iteratively mutating and crossing over parent structures to minimize deformation energy.

Datasets

• MatBenchbandgap — total 5000; splits: reference_pool (5000)

Metrics

• deformation energy (primary) — range: other
  • Average deformation energy (Ed) of the top-k generated structures, evaluated by CHGNet. Lower Ed indicates higher stability.

Input / output format

Input: Two parent crystal structures with their fitness values (Ed), sampled from the current population.

Output: Five new crystal structures generated via mutation or crossover.

Scoring recipe

def score(predictions, gold):
    energies = [chgnet_energy(pred) for pred in predictions]
    top_k_avg = average(energies[:k])
    return top_k_avg  # Lower is better

Common pitfalls

• Early stopping criteria based on mean fitness improvement thresholds may terminate runs prematurely if the landscape is noisy.
• Merging parent and child evaluations without proper ranking can bias subsequent generations toward local optima.

Evidence (verbatim from paper)

Each experiment began with an initial population of 100 groups of parents... randomly seeded from the reference pool, which is composed with 5,000 known stable structures from MatBenchbandgap dataset with lowest deformation energy evaluated by CHGNet.

Citation

@misc{song2025evaluating,
  title={Evaluating Large Language Models in Scientific Discovery},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2512.15567}
}

• arXiv: 2512.15567