Medical Mcqa Eval

medical-mcqa-eval

Generating multiple-choice questions for medical question answering with distractors and cue-masking — Sileo et al. (2023) (arXiv:2303.07069, 2023)

What this evaluates

Evaluates a model's ability to answer medical multiple-choice questions by selecting the correct option from a set of distractors, including natural differential diagnoses. It tests both internal knowledge retrieval and the impact of synthetic pretraining with cue-masking strategies.

Datasets

• MedQA-USMLE — total ?; splits: test (-1)
• MedMCQA — total ?; splits: train (-1)
• DBPedia — total 3446; splits: test (3446)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard multiple-choice accuracy: the proportion of instances where the model's predicted option (argmax of softmax probabilities) matches the gold correct option.
• precision@3 / recall@3 — range: [0, 1]
  • For distractor prediction: precision@3 is the fraction of top-3 retrieved differential diagnoses that are correct; recall@3 is the fraction of all correct differential diagnoses retrieved in the top-3.

Input / output format

Input: Question text concatenated with each answer option to compute logit scores. Optionally, the top 10 retrieved Wikipedia passages are concatenated to the question for knowledge-augmented evaluation.

Output: Logit scores for each option, passed through softmax to yield probabilities. The model selects the option with the highest probability as its prediction.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_pr_at_k(retrieved, relevant, k=3):
    top_k = retrieved[:k]
    precision = len(set(top_k) & set(relevant)) / k
    recall = len(set(top_k) & set(relevant)) / len(relevant) if relevant else 0
    return precision, recall

Common pitfalls

• Extraneous cue leakage occurs if masking strategies inadvertently hide tokens that signal the correct answer, artificially lowering performance.
• Models may rely on external knowledge retrieval (BM25) rather than internalized medical knowledge, making results sensitive to the retrieval setup.
• Probability-matching masking requires careful handling of negative examples to avoid masking tokens that would otherwise indicate the correct answer.

Evidence (verbatim from paper)

We use a multiple-choice-question answering setup (we predict logit scores for each option by concatenating the question and the option, then use a softmax and optimize the likelihood of the correct option). Table 2 shows the test accuracy of BioLinkBERT fine-tuned on WikiMedQA then on various datasets, compared to the task-specific state-of-the-art.

Citation

@misc{sileo2023generating,
  title={Generating multiple-choice questions for medical question answering with distractors and cue-masking},
  author={Sileo et al. (2023)},
  year={2023},
  note={arXiv:2303.07069}
}

• arXiv: 2303.07069