Molbench Eval

molbench-eval

MolClaw: An Autonomous Agent with Hierarchical Skills for Drug Molecule Evaluation, Screening, and Optimization — Zhang et al. (2026) (arXiv:2604.21937, 2026)

What this evaluates

Evaluates autonomous AI agents' ability to execute complex, multi-step drug discovery workflows, including molecular screening (property filtering, binding affinity comparison, docking) and molecular optimization (structural editing, physicochemical property improvement).

Datasets

• MolBench — total ?; splits: MolBench-MS (112), MolBench-MO (39); repo https://github.com/InternScience/MolClaw

Metrics

• property filtering accuracy — range: percent
  • Percentage of queries where the agent correctly applies threshold-based constraints to physicochemical descriptors.
• property filtering F1 score — range: percent
  • Harmonic mean of precision and recall for property filtering tasks.
• binding affinity accuracy — range: percent
  • Percentage of molecular pairs correctly ranked or classified by binding affinity using a validated multi-step docking pipeline.
• docking hit count — range: [0, 3]
  • Average number of valid docking hits returned in the top 3 predictions per target.
• molecule editing accuracy — range: percent
  • Percentage of tasks where the agent correctly applies specified chemical transformations (e.g., functional group replacement, scaffold hopping).
• optimization delta — range: other
  • Mean improvement score across three physicochemical properties (QED, LogP, LogS) after iterative structural modification and verification.
• optimization success rate (primary) — range: percent
  • Percentage of optimization tasks where all target physicochemical constraints are simultaneously satisfied after iterative refinement.

Input / output format

Input: Molecular structures (SMILES/3D coordinates), target protein sequences/structures, physicochemical constraints, and optimization objectives.

Output: Generated molecular structures (SMILES/3D), computed property values, docking scores, and binary success indicators.

Scoring recipe

def evaluate_molbench(predictions, gold):
    scores = {}
    for task in ['prop_filter', 'bind_aff', 'edit']:
        scores[f'{task}_acc'] = sum(p == g for p, g in zip(predictions[task], gold[task])) / len(gold)
    scores['prop_filter_f1'] = f1_score(gold['prop_filter'], predictions['prop_filter'])
    scores['docking_hits'] = mean(len([s for s in pred[:3] if s > threshold]) for pred in predictions['docking'])
    scores['opt_delta'] = mean(pred['delta'] for pred in predictions['opt'])
    scores['opt_sr'] = sum(pred['success'] for pred in predictions['opt']) / len(gold)
    return scores

Common pitfalls

• LLMs cannot reliably compute molecular descriptors through reasoning alone; tool access is the primary bottleneck for simple tasks.
• Vanilla agent frameworks lack structured multi-step pipelines, causing performance plateaus on complex tasks despite strong code generation ability.
• Optimization delta and success rate capture different aspects of iterative refinement; a high delta does not guarantee meeting all constraints.
• Statistical significance depends on task complexity; performance gains over near-ceiling baselines are often non-significant.

Evidence (verbatim from paper)

Across all seven metrics spanning the MolBench-MS and MolBench-MO benchmarks, MolClaw on Claude Code (MolClaw-CC) ranked first or tied for first on every metric, achieving sole best performance on four (binding affinity accuracy, docking hit count, molecule editing accuracy, and optimization delta) and tied best on three (property filtering accuracy and F1 score, optimization success rate)

Citation

@misc{zhang2026molclaw,
  title={MolClaw: An Autonomous Agent with Hierarchical Skills for Drug Molecule Evaluation, Screening, and Optimization},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.21937}
}

• arXiv: 2604.21937