Drugpc Eval

drugpc-eval

TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Tools — Gao et al. (2025) (arXiv:2503.10970, 2025)

What this evaluates

Probes multi-step therapeutic reasoning and tool-use for drug-related questions, including interactions, contraindications, and patient-specific treatment strategies. It tests the model's ability to dynamically select biomedical tools, retrieve verified knowledge, and generate evidence-grounded answers.

Datasets

• DrugPC — total 3168; splits: test (3168)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions. Evaluated in two settings: multiple-choice (selecting from 4–5 predefined options) and open-ended (generating a free-text response, then mapping it to the correct option).

Input / output format

Input: Therapeutic questions about drugs, diseases, or patient populations. In multiple-choice mode: question plus 4–5 answer options. In open-ended mode: question only.

Output: A multi-step reasoning trace (thought process, function calls, tool feedback) followed by a final answer. For evaluation, the final answer is matched to the correct option.

Scoring recipe

correct = 0
for q, gold_opt in dataset:
    if setting == 'multiple_choice':
        pred = model.generate(q, options)
    else:
        pred_text = model.generate(q)
        pred = extract_option_from_text(pred_text, options)
    if pred == gold_opt:
        correct += 1
return correct / len(dataset)

Common pitfalls

• Data leakage: Pretraining data often contains drugs approved before 2024, so the benchmark restricts evaluation to 2024 FDA-approved drugs to ensure fair testing of real-time tool use.
• Open-ended vs. multiple-choice gap: Models heavily rely on answer choices in MC format; switching to open-ended causes significant accuracy drops for baselines, revealing true reasoning capability.
• Tool execution verification: Accuracy alone does not capture whether the model actually invoked the correct tools or hallucinated the answer; reasoning traces must be inspected for valid tool calls.

Evidence (verbatim from paper)

We construct the DrugPC (Drug Prescribing Card) benchmark to evaluate TxAgent’s performance in drug reasoning. DrugPC includes 3,168 questions spanning 11 tasks: drug overview, ingredients, warnings and safety, dependence and abuse, dosage and administration, use in specific populations, pharmacology, clinical information, nonclinical toxicology, patient-focused information, and storage and supply. We evaluate models in two settings: multiple-choice, where the model selects the correct answer from given options, and open-ended, where the model generates responses without predefined choices. In the multiple-choice setting, TxAgent achieves 93.8% accuracy... In the open-ended setting, TxAgent maintains 92.1% accuracy...

Citation

@misc{gao2025txagent,
  title={TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Tools},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2503.10970}
}

• arXiv: 2503.10970