Drugcareqa Eval

drugcareqa-eval

MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision — Zheng et al. (2025) (arXiv:2510.10461, 2025)

What this evaluates

Evaluates an AI system's ability to perform integrated clinical decision-making by simulating real-world online medical consultations. It probes the model's capacity to reason through patient symptoms, generate accurate diagnoses, and recommend appropriate medications within a unified workflow.

Datasets

• DrugCareQA — total 2700; splits: test (-1)

Metrics

• diagnostic and medication recommendation accuracy (primary) — range: percent
  • Percentage of instances where the model's predicted diagnosis and medication recommendation both exactly match the expert-verified ground truth. Calculated as (correct predictions / total instances) * 100.

Input / output format

Input: Patient chief complaint or multi-turn dialogue describing symptoms, potentially including demographic and clinical presentation details.

Output: A structured response containing a final diagnosis and a specific medication recommendation.

Scoring recipe

def calculate_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if pred['diagnosis'] == gold_item['diagnosis'] and pred['medication'] == gold_item['medication']:
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Models may optimize for diagnosis only and ignore medication recommendation, or vice versa, since the benchmark requires both to be correct for a positive score.
• The dataset uses real-world patient dialogues with ambiguous symptom descriptions, which can cause models to hallucinate diagnoses if they rely too heavily on rigid multiple-choice formats from other benchmarks.
• Medication recommendations must match the exact drug names specified in the expert-verified ground truth, as similar drug distractors are intentionally included to test clinical logic.

Evidence (verbatim from paper)

We integrate diagnostic reasoning with medication selection into a unified evaluation framework. The dataset encompasses 2,700 annotated cases across seven clinical departments with dual quality verification combining knowledge base validation and expert review. Models are evaluated on diagnostic and medication recommendation accuracy, achieving 67.58% on DrugCareQA—a benchmark of 2,700 real-world patient-doctor dialogues that unifies diagnosis and treatment in a clinically realistic workflow.

Citation

@misc{zheng2025medcoact,
  title={MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.10461}
}

• arXiv: 2510.10461