Retrosynthesis Solve Rate Eval

retrosynthesis-solve-rate-eval

Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)

What this evaluates

Evaluates an LLM's ability to generate valid chemical synthesis pathways for target molecules given reference routes and iterative feedback.

Datasets

• Pistachio Hard — total 100; splits: test (100)

Metrics

• solve rate (primary) — range: percent
  • Percentage of target molecules for which the LLM produces a valid synthesis route within 100 queries. Validity requires RDKit-parsable molecules, commercially available precursors, exact-match reaction templates, and correct data format.

Input / output format

Input: Target molecule SMILES, up to 3 reference synthesis routes retrieved via Tanimoto similarity, and iterative feedback on route validity.

Output: A proposed synthesis route in a prescribed data format, or a revised route if the first step fails.

Scoring recipe

def score(predictions, gold):
    valid_count = 0
    for pred in predictions:
        if check_molecule_validity(pred) and check_reaction_validity(pred) and check_route_format(pred):
            valid_count += 1
    return (valid_count / 100) * 100

Common pitfalls

• Models may output routes that violate the prescribed data format, causing automatic rejection despite chemically valid steps.
• Reaction templates not in the database trigger fallback logic that may mask true model capability if not tracked separately.

Evidence (verbatim from paper)

Using the Pistachio Hard benchmark set of 100 molecule targets, we evaluated the solve rate using multiple LLMs and compared to common methods spanning MCTS and Retro* search... The evaluation runs the LLM framework on prescribed sets of target molecules and reports the number of molecules that return a valid synthesis route, within 100 LLM queries (i.e. the solve rate).

Citation

@misc{song2025evaluating,
  title={Evaluating Large Language Models in Scientific Discovery},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2512.15567}
}

• arXiv: 2512.15567