Medical QA Explanation Eval

medical-qa-explanation-eval

Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions — Chen et al. (2024) (arXiv:2402.18060, 2024)

What this evaluates

Evaluates large language models' ability to answer challenging medical multiple-choice questions and generate step-by-step clinical reasoning explanations. It probes both factual accuracy in clinical decision-making and the quality of model-generated rationales compared to expert-written references.

Datasets

• JAMA Clinical Challenge — total ?; splits: test (-1)
• Medbullets — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed as the proportion of model predictions that exactly match the ground truth answer choice.
• ROUGE-L — range: [0, 1]
  • Computes the longest common subsequence overlap between model-generated explanations and reference (gold) explanations to measure surface-form similarity.
• BERTScore — range: [0, 1]
  • Measures semantic similarity between model-generated and gold explanations using contextual embeddings from BERT.
• BARTScore+ — range: [0, 1]
  • Finetuned variant of BARTScore using CNN data to compute semantic similarity between generated and reference explanations.
• BARTScore++ — range: [0, 1]
  • Finetuned variant of BARTScore using CNN and paraphrase data to compute semantic similarity between generated and reference explanations.

Input / output format

Input: Clinical case description, question, and multiple-choice answer options (X).

Output: Model-generated answer choice (Y), optionally preceded by step-by-step reasoning (R), or an explanation (R) given the input and gold answer (Y*).

Scoring recipe

def compute_metrics(predictions, gold_answers, explanations, gold_explanations):
    acc = sum(1 for p, g in zip(predictions, gold_answers) if p == g) / len(predictions)
    rouge_l = rouge_l_score(explanations, gold_explanations)
    bert = bertscore_f1(explanations, gold_explanations)
    bart_plus = bartscore_score(explanations, gold_explanations, variant='CNN')
    bart_plus_plus = bartscore_score(explanations, gold_explanations, variant='CNN+Para')
    return {'accuracy': acc, 'ROUGE-L': rouge_l, 'BERTScore': bert, 'BARTScore+': bart_plus, 'BARTScore++': bart_plus_plus}

Common pitfalls

• The evaluation uses three distinct prompting strategies (X→Y, X→RY, XY*→R) that drastically change the model's output format and reasoning process; results must be reported per strategy rather than aggregated.
• Explanation metrics (ROUGE-L, BERTScore, BARTScore variants) measure text similarity to gold references rather than clinical correctness or logical validity, which may not align with expert medical judgment.

Evidence (verbatim from paper)

We use accuracy (prediction compared to ground truth) to evaluate the predictions of each model. We explore several methods to evaluate model-generated explanations: ROUGE-L, BERTScore, and two finetuned variants of BARTScore—BARTScore+CNN (BARTScore+) and BARTScore+CNN+Para (BARTScore++). These metrics compute the surface-form or semantic similarities between model-generated explanations and reference (gold) explanations.

Citation

@misc{chen2024benchmarking,
  title={Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2402.18060}
}

• arXiv: 2402.18060