Medmcqa Eval

medmcqa-eval

MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering — Pal et al. (2022) (arXiv:2203.14371, 2022)

What this evaluates

Evaluates a model's ability to answer medical multiple-choice questions, testing both domain-specific knowledge retrieval and deep medical reasoning capabilities.

Datasets

• MedMCQA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard multiple-choice accuracy: the proportion of questions where the model's predicted option exactly matches the ground-truth correct option.

Input / output format

Input: Concatenated sequence: [CLS] Context [SEP] Question [SEP] Option [SEP] (with context) or [CLS] Question [SEP] Option [SEP] (without context). Passages are truncated to 250 tokens.

Output: A single predicted option index selected via softmax over the [CLS] token's final hidden state.

Scoring recipe

correct = 0
for q, options, gold_idx in dataset:
    logits = model(input_ids)
    pred_idx = argmax(logits[0])
    if pred_idx == gold_idx:
        correct += 1
accuracy = correct / len(dataset)

Common pitfalls

• Retrieval errors from the dense passage retriever are often conflated with the reader model's reasoning failures.
• Truncating medical passages to 250 tokens may discard critical clinical details needed to answer the question.
• Evaluating only on the test set without reporting calibration or confidence scores limits understanding of model reliability in high-stakes medical domains.

Evidence (verbatim from paper)

A linear layer with softmax is applied over the output of the [CLS] token of the encoder. This is to select the most appropriate option for a question and context pair. ... The model checkpoint with the highest validation score in the 5 epochs was selected and used to evaluate the Test Set.

Citation

@misc{pal2022medmcqa,
  title={MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering},
  author={Pal et al. (2022)},
  year={2022},
  note={arXiv:2203.14371}
}

• arXiv: 2203.14371