Frogent Drug Design Eval

frogent-drug-design-eval

FROGENT: An End-to-End Full-process Drug Design Agent — Pan et al. (2025) (arXiv:2508.10760, 2025)

What this evaluates

Evaluates an end-to-end agentic framework for small-molecule drug design across eight benchmarks spanning the full discovery pipeline, from target identification and knowledge retrieval to virtual screening, interaction profiling, de novo design, and retrosynthetic planning.

Datasets

• Humanity’s Last Exam (HLE) — total ?; splits: test (-1)
• UniProt — total ?; splits: test (-1)
• Open Targets Platform — total ?; splits: test (-1)
• ADMETLab 3.0 — total ?; splits: test (-1)
• DAVIS — total ?; splits: test (-1)
• PLIP — total ?; splits: test (-1)
• CrossDocked — total ?; splits: test (-1)
• USPTO-50k — total ?; splits: test (-1)
• PaRoute — total ?; splits: test (-1)

Metrics

• score (primary) — range: other
  • Task-specific aggregate score reported per benchmark. The exact mathematical formula, normalization, and scale are not specified in the text; values range from 6 to 95 across different tasks.

Input / output format

Input: Varies by task: biomedical knowledge queries, protein identifiers/sequences for active molecule retrieval, disease-target pairs, molecular property inputs, protein-ligand complexes for screening/profiling, protein pocket structures for de novo design, and target molecules for retrosynthesis.

Output: Task-dependent predictions: knowledge answers, molecule identifiers/structures, property values, ranked candidate molecules, interaction annotations, generated molecular structures, and retrosynthetic reaction pathways.

Scoring recipe

# Note: Exact scoring formulas per task are not specified in the text.
# Reported values are raw task scores.
def compute_score(predictions, gold, task_type):
    if task_type in ['knowledge', 'retrieval', 'validation']:
        return exact_match_or_accuracy(predictions, gold)
    if task_type == 'property':
        return regression_or_classification_score(predictions, gold)
    if task_type in ['screening', 'interaction']:
        return hit_rate_or_rank(predictions, gold)
    if task_type == 'design':
        return predicted_property_score(predictions, gold)
    if task_type == 'retrosynthesis':
        return reaction_accuracy(predictions, gold)
    return 0

Common pitfalls

• Raw scores are reported without specifying the metric formula, normalization, or scale for each of the eight tasks.
• Baselines lack domain-specific tool access, making direct performance comparisons dependent on the agent's ability to parse specialized database schemas rather than pure reasoning.
• Scores vary widely across tasks (e.g., 6 to 95), indicating task-specific scoring functions rather than a unified metric.

Evidence (verbatim from paper)

Frogent achieved a score of 6, significantly outperforming the best baseline. The modest performance of the base LLMs on this task was expected, as it relies on their internal knowledge. However, this superiority was starkly evident in tasks requiring agents to reason over complex structured data. When tasked with retrieving known active molecules for given proteins from UniProt, which is a common starting point for lead optimization campaigns. Frogent achieved a near-perfect score of 83, while all baseline agents completely failed. This result highlights a critical limitation of general-purpose agents. Without the domain-specific knowledge of how to query and parse the outputs from specialized scientific databases and interfaces, they are incapable of navigating the complex schemas of essential scientific repositories. Similarly, in a disease-target validation task using data from the Open Targets Platform, which is necessary for de-risking the entire project by confirming the therapeutic hypothesis, Frogent scored 95, surpassing the strongest baseline and demonstrating its robust data retrieval capabilities.

Citation

@misc{pan2025frogent,
  title={FROGENT: An End-to-End Full-process Drug Design Agent},
  author={Pan et al. (2025)},
  year={2025},
  note={arXiv:2508.10760}
}

• arXiv: 2508.10760