Chemcotb Eval

chemcotb-eval

Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations — Hao Li et al. (2025) (arXiv:2505.21318, 2025)

What this evaluates

Evaluates large language models' ability to perform stepwise chemical reasoning and molecular structure manipulation. It probes capabilities in molecular understanding, functional group/ring recognition, scaffold extraction, and SMILES-based molecule editing and reaction prediction.

Datasets

• ChemCoTBench — total 1495; splits: test (-1)

Metrics

• MAE — range: other
  • Mean Absolute Error used for counting functional groups and rings.
• Tanimoto similarity — range: [0, 1]
  • Measures structural overlap between predicted and reference Murcko scaffolds.
• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted answers for binary, multiple-choice, and ring-system identification tasks.
• Pass@1 — range: [0, 1]
  • Proportion of editing tasks where the single generated molecule satisfies all structural instructions.
• Top-1 accuracy — range: [0, 1]
  • Exact match rate for generated SMILES strings in reaction prediction tasks.
• FTS — range: [0, 1]
  • Fingerprint-based similarity using Morgan, MACCS, or RDKit fingerprints to measure structural similarity between generated and reference molecules.

Input / output format

Input: Source SMILES string, task instruction (e.g., optimization target, editing rule, reaction condition), and sometimes reference molecules or multiple-choice options.

Output: Edited or generated SMILES string, predicted numerical property value, or selected multiple-choice option.

Scoring recipe

def score(prediction, gold, task_type):
    if task_type == 'counting':
        return abs(int(prediction) - int(gold))  # MAE
    elif task_type == 'scaffold':
        return tanimoto_similarity(mol_from_smiles(prediction), mol_from_smiles(gold))
    elif task_type == 'editing':
        return 1.0 if passes_chemical_constraints(prediction, gold) else 0.0  # Pass@1
    elif task_type == 'reaction':
        exact = 1.0 if prediction == gold else 0.0  # Top-1
        sim = max(fingerprint_similarity(prediction, gold, fp) for fp in ['morgan', 'maccs', 'rdkit'])  # FTS
        return exact, sim
    else:
        return 1.0 if prediction == gold else 0.0  # accuracy

Common pitfalls

• SMILES equivalence requires canonicalization or graph isomorphism checking rather than simple string comparison.
• FTS similarity scores depend heavily on the chosen fingerprint type (Morgan, MACCS, RDKit) and radius/length parameters.
• Pass@1 for molecule editing implicitly requires valid chemical valency and ring closure checks, not just instruction matching.

Evidence (verbatim from paper)

For understanding tasks, functional group (FG) and ring recognition are treated as counting problems, with mean absolute error (MAE) used to measure precision. Scaffold-level understanding includes extracting Murcko scaffolds, evaluated by Tanimoto similarity, and identifying whether complex ring systems are present, evaluated by accuracy. The SMILES equivalence task is formulated as a binary decision problem, determining whether the target and source SMILES represent the same molecule, and is also evaluated using accuracy. For molecule editing, we use Pass@1 to assess whether the edited molecule meets the instructions. Mechanism route selection is framed as a multiple-choice task and evaluated by accuracy. Other reaction tasks are modeled as SMILES generation problems, where evaluation is based on both Top-1 accuracy and fingerprint-based similarity (FTS), using Morgan, MACCS, and RDKit fingerprints to reflect correctness and structural similarity.

Citation

@misc{li2025beyondchemicalqa,
  title={Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations},
  author={Hao Li et al. (2025)},
  year={2025},
  note={arXiv:2505.21318}
}

• arXiv: 2505.21318