Chemllmbench Eval

chemllmbench-eval

ChemHAS: Hierarchical Agent Stacking for Enhancing Chemistry Tools — Zhucong Li et al. (arXiv:2505.21569, 2025)

What this evaluates

Evaluates LLM-based chemistry agents on four core computational chemistry tasks: text-to-molecule generation, molecule-to-text captioning, molecular property prediction, and reaction product prediction. It probes the model's ability to translate between chemical language and structures, predict physicochemical properties, and correct tool errors via hierarchical agent stacking.

Datasets

• ChemLLMBench — total 400; splits: test (400)

Metrics

• Exact match — range: [0, 1]
  • Binary score: 1 if the predicted SMILES string exactly matches the gold SMILES string, 0 otherwise.
• BLEU — range: [0, 1]
  • N-gram precision score measuring overlap between predicted and reference text or SMILES sequences.
• Levenshtein distance — range: other
  • Minimum number of single-character edits (insertions, deletions, substitutions) required to change one SMILES string into the other.
• Validity — range: [0, 1]
  • Fraction of predicted SMILES strings that are chemically valid according to standard chemical parsers (e.g., RDKit).
• FTS (MACCS/RDK/Morgan) — range: [0, 1]
  • Fingerprint Tanimoto Similarity calculated between predicted and reference molecules using MACCS, RDKit, or Morgan fingerprints.
• BLEU-2/4 — range: [0, 1]
  • N-gram precision scores for 2-gram and 4-gram overlaps between generated captions and reference descriptions.
• ROUGE-1/2/L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation scores measuring n-gram and longest common subsequence overlap for captions.
• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted molecular properties (binary or multi-class) out of total instances.

Input / output format

Input: Per task: (1) Text description of a molecule, (2) SMILES string, (3) Molecular structure/SMILES for property prediction, (4) Reactant SMILES for reaction prediction.

Output: Per task: (1) Predicted SMILES string, (2) Natural language caption, (3) Predicted property label/value, (4) Predicted product SMILES string.

Scoring recipe

def compute_metrics(pred, gold):
    exact = 1.0 if pred == gold else 0.0
    bleu = sacrebleu.corpus_bleu([pred], [[gold]]).score
    valid = 1.0 if Chem.MolFromSmiles(pred) is not None else 0.0
    if valid:
        mol_pred = Chem.MolFromSmiles(pred)
        mol_gold = Chem.MolFromSmiles(gold)
        fts = TanimotoSimilarity(Chem.GetMorganFingerprintAsBitVect(mol_pred, 2),
                                 Chem.GetMorganFingerprintAsBitVect(mol_gold, 2))
    else:
        fts = 0.0
    acc = 1.0 if pred == gold else 0.0
    return exact, bleu, valid, fts, acc

Common pitfalls

• Evaluating LLM-based generalist models in 0-shot while comparing against fine-tuned task-specific specialists creates an unfair baseline comparison.
• SMILES validity is a hard constraint; invalid predictions break downstream chemical similarity (FTS) calculations and must be handled explicitly.
• Validation sets are sampled from external datasets (ChEBI-20-MM, USPTO-MIT, MoleculeNet) excluding their official test sets, risking data leakage if overlap isn't strictly verified.

Evidence (verbatim from paper)

The first set of metrics measures the text-based similarity between the predicted SMILES and the gold standard SMILES, including exact match, BLEU, and Levenshtein distance. The second set of metrics assesses the chemical similarity between the predicted molecules and the reference molecules, encompassing the validity of the predicted SMILES and the FTS (Fingerprint Tanimoto Similarity), calculated based on MACCS, RDK, and Morgan.

Citation

@misc{li2025chemhas,
  title={ChemHAS: Hierarchical Agent Stacking for Enhancing Chemistry Tools},
  author={Zhucong Li et al.},
  year={2025},
  note={arXiv:2505.21569}
}

• arXiv: 2505.21569