qhjqhj00 tmc-optimization-eval
Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)
What this evaluates
Tests an LLM's ability to iteratively design transition metal complexes (TMCs) by maximizing specific properties (polarisability) or expanding multi-objective Pareto frontiers.
Datasets
- Pd(II) square planar complex space — total 1370000; splits: search_space (1370000)
Metrics
Pareto frontier quality(primary) — range: percent | other- Binary success rate for finding the global optimum in polarisability maximization, and qualitative/quantitative coverage of the Pareto frontier for multi-objective expansion (HOMO-LUMO gap > 4 eV, polarisability > 400 a.u.).
Input / output format
Input: Pool of 50 ligands (SMILES, IDs, charges, connecting atoms), 20 initial TMCs with properties, and natural language design objectives.
Output: A new set of TMCs with proposed structures and properties.
Scoring recipe
def score(predictions, gold):
if task == 'max_polarisability':
return 1.0 if find_optimal(predictions) else 0.0
elif task == 'pareto_frontier':
return count_pareto_dominated(predictions, gold) / total_possible
return 0.0
Common pitfalls
- Random seed selection significantly impacts the initial sampling and subsequent Pareto frontiers, requiring multiple seeds for reliable evaluation.
- Validation steps (charge constraints, geometry optimization, connectivity) can fail silently if not strictly enforced, leading to invalid complexes being counted.
Evidence (verbatim from paper)
In the first task of proposing TMCs with maximized polarisability, gpt-5, deepseek-R1, and claude-sonnet-4.5 successfully finds the optimal solution in the space of 1.37M TMCs at all five random seeds... A similar trend is observed when models are asked to expand the Pareto frontiers...
Citation
@misc{song2025evaluating,
title={Evaluating Large Language Models in Scientific Discovery},
author={Song et al. (2025)},
year={2025},
note={arXiv:2512.15567}
}
- arXiv: 2512.15567