Medexqa Eval

medexqa-eval

MedExQA: Medical Question Answering Benchmark with Multiple Explanations — Kim et al. (2024) (arXiv:2406.06331, 2024)

What this evaluates

Evaluates medical language models on multiple-choice question answering and the generation of clinically relevant explanations. It probes the model's ability to perform clinical reasoning, avoid hallucinations, and produce coherent, accurate rationales aligned with medical domain knowledge.

Datasets

• MedExQA — total ?; splits: test (-1), dev (-1)

Metrics

• Classification Accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered multiple-choice questions. For generative models, the next token with the highest logit is selected, or string matching is used to find the answer choice/letter in the generated text. Score is 1 if match, 0 otherwise.
• BLEU — range: [0, 1]
  • Geometric mean of precision scores of generated explanations compared to reference explanations based on n-gram matches.
• ROUGE-L — range: [0, 1]
  • Measures similarity based on the longest common subsequence between generated and reference explanations, combining precision and recall.
• METEOR — range: [0, 1]
  • Considers semantic similarity and lexical variations using WordNet alignment between generated and reference texts.
• BERTScore — range: [0, 1]
  • Uses contextual embeddings (SciBERT) to capture semantic nuances between generated and reference explanations.
• Human Evaluation Score — range: [0, 1]
  • Annotators assign 0 for incorrect/irrelevant/missing explanations, 0.5 for correct answer but incorrect/incomplete explanation, and 1.0 for both correct answer and explanation.

Input / output format

Input: Multiple-choice medical questions with four options, plus reference explanations for generation tasks.

Output: For classification: the predicted answer choice (letter or exact phrase). For explanation generation: a textual rationale explaining the correct answer.

Scoring recipe

def evaluate_medexqa(predictions, golds):
    acc_scores = []
    lex_scores = []
    for pred, gold in zip(predictions, golds):
        is_correct = (pred['answer'] == gold['answer']) or (gold['answer'] in pred['text'])
        acc_scores.append(1.0 if is_correct else 0.0)
        lex_scores.append(bertscore_f1(pred['explanation'], gold['explanation']))
    combined = [l if a == 1.0 else 0.0 for a, l in zip(acc_scores, lex_scores)]
    return {
        'accuracy': sum(acc_scores) / len(acc_scores),
        'bertscore': sum(lex_scores) / len(lex_scores),
        'combined': sum(combined) / len(combined)
    }

Common pitfalls

• Logit-based accuracy cannot be computed for closed-source APIs (e.g., GPT-3.5/4), requiring fallback to string matching which may miss paraphrased answers.
• Reporting lexical metrics (BLEU/ROUGE/BERTScore) without filtering out incorrect answers inflates performance, as the protocol mandates a score of 0 for wrong answers.
• Human evaluation was only performed on a small development subset (5 samples per specialty), so automated metrics are the primary reported benchmarks.

Evidence (verbatim from paper)

The quality of generated explanations is further assessed using a combination of general lexical metrics. BLEU measures the geometric mean of precision scores of the generated explanations compared to reference explanations based on n-gram matches. ROUGE assesses the similarity between generated and reference explanations, with ROUGE-L, providing a score that combines precision and recall based on the longest common subsequence. METEOR considers the semantic similarity and lexical variations with WordNet. BERTScore uses contextual embeddings, scibert embedding for our work, to capture nuances in the semantics of the explanations. We propose an enhanced methodology for evaluating models’ understanding of medical domain knowledge by incorporating classification accuracy based on string matches into calculating these metrics. We assign a score of 0 to responses with incorrect answers based on string-matching classification results.

Citation

@misc{kim2024medexqa,
  title={MedExQA: Medical Question Answering Benchmark with Multiple Explanations},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2406.06331}
}

• arXiv: 2406.06331