Ehrnoteqa Eval

ehrnoteqa-eval

EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries — Kweon et al. (2024) (arXiv:2402.16040, 2024)

What this evaluates

Evaluates large language models' ability to perform patient-specific clinical reasoning by synthesizing information from multiple electronic health record (EHR) discharge summaries to answer medical questions. It specifically tests multi-document clinical analysis and automated medical model evaluation using structured multi-choice or free-text formats.

Datasets

• EHRNoteQA — total 962; splits: test (962); repo https://github.com/ji-youn-kim/EHRNoteQA

Metrics

• score (primary) — range: [0, 100]
  • Models receive 1 point for a correct answer and 0 for incorrect. The final score is the average points across all questions, normalized to a 100-point scale.

Input / output format

Input: Patient discharge summaries (clinical notes), a clinical question, and multiple-choice answer options (for multi-choice evaluation) or just the question (for free-text evaluation).

Output: Model-generated text response, either selecting an answer choice letter/sequence or providing a free-text explanation.

Scoring recipe

def compute_score(predictions, gold_answers):
    total_score = 0.0
    for pred, gold in zip(predictions, gold_answers):
        iter_results = []
        for _ in range(5):
            judge = call_gpt4turbo(f"Evaluate if prediction '{pred}' matches gold '{gold}'.")
            iter_results.append(1 if judge == 'correct' else 0)
        total_score += sum(iter_results) / 5
    return (total_score / len(predictions)) * 100

Common pitfalls

• Probability-based scoring (log-probs) is infeasible because multi-note EHR contexts exceed the context windows of most models, making few-shot prompting impractical.
• Free-text evaluation yields high variance in model rankings and scores compared to multi-choice, making it unreliable for automated benchmarking.
• GPT-4-turbo automated scoring can be inconsistent for ambiguous responses, requiring repeated evaluations and averaging to stabilize results.

Evidence (verbatim from paper)

For each model, correct outputs were assigned of 1 point, while incorrect responses were given 0 points. Across 962 questions, the scores of each model were normalized to a 100-point scale for comparison.

Citation

@misc{kweon2024ehrnoteqa,
  title={EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries},
  author={Kweon et al. (2024)},
  year={2024},
  note={arXiv:2402.16040}
}

• arXiv: 2402.16040