qhjqhj00 molecule-optimization-auc-eval
Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)
What this evaluates
Assesses an LLM's capability to iteratively optimize molecular structures for specific biological targets (JNK3, GSK3β) using evolutionary search guided by generative prompts.
Datasets
- ZINC — total ?; splits: reference_pool (-1)
Metrics
AUC_top-k(primary) — range: [0, 1]- Area under the curve of the average fitness of the top-10 molecules versus the number of oracle calls (capped at 10,000). Captures both optimization quality and sample efficiency.
Input / output format
Input: One or two parent molecules with their fitness scores, sampled from the current population.
Output: A new molecule SMILES string generated via mutation or crossover.
Scoring recipe
def score(predictions, gold):
fitness_history = []
for call in range(10000):
top10_avg = average_fitness(top_k(predictions, k=10))
fitness_history.append(top10_avg)
return trapezoid_area(fitness_history) # AUC_top-k
Common pitfalls
- Decoding temperature mismatches (e.g., GPT-5 forced to temp=1.0 vs others at 0.8) can artificially inflate or deflate AUC scores.
- High duplication rates in generated molecules reduce effective exploration, penalizing AUC even if final top-k quality is strong.
Evidence (verbatim from paper)
Methods were compared using the area under the curve of the top-k average objective versus the number of oracle calls (AUC_top-k) with k = 10, which jointly captures optimization quality and sample efficiency.
Citation
@misc{song2025evaluating,
title={Evaluating Large Language Models in Scientific Discovery},
author={Song et al. (2025)},
year={2025},
note={arXiv:2512.15567}
}
- arXiv: 2512.15567