Medagents Medical QA Eval

medagents-medical-qa-eval

MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning — Tang et al. (2023) (arXiv:2311.10537, 2023)

What this evaluates

Evaluates zero-shot medical reasoning and multiple-choice question answering capabilities of LLMs using a training-free multi-agent collaboration framework. It probes the model's ability to simulate domain expert role-playing and reach consensus without retrieval-augmented generation.

Datasets

• MedQA — total ?; splits: test (300)
• MedMCQA — total ?; splits: test (300)
• PubMedQA — total ?; splits: test (300)
• MMLU Anatomy — total ?; splits: test (300)
• MMLU Clinical Knowledge — total ?; splits: test (300)
• MMLU College Medicine — total ?; splits: test (300)
• MMLU Medical Genetics — total ?; splits: test (300)
• MMLU Professional Medicine — total ?; splits: test (300)
• MMLU College Biology — total ?; splits: test (300)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions out of the total evaluated instances.

Input / output format

Input: Multiple-choice medical question with 3 or 4 options.

Output: Selected answer option (e.g., A, B, C, or D).

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
return (correct / len(gold_labels)) * 100

Common pitfalls

• Chain-of-thought prompting can cause hallucinations and degrade performance in medical domains.
• Cost and inference time vary significantly with the number of agents and iterations.
• Zero-shot settings may underperform compared to few-shot baselines without careful prompt engineering.

Evidence (verbatim from paper)

We evaluate our MEDAGENTS framework on three benchmark datasets MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022), and PubMedQA (Jin et al., 2019), as well as six subtasks most relevant to the medical domain from MMLU datasets (Hendrycks et al., 2020) including anatomy, clinical knowledge, college medicine, medical genetics, professional medicine, and college biology. We randomly sample 300 examples for each dataset and conduct experiments on them.

Citation

@misc{tang2023medagents,
  title={MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning},
  author={Tang et al. (2023)},
  year={2023},
  note={arXiv:2311.10537}
}

• arXiv: 2311.10537