qhjqhj00 drbench-eval
DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing — Gao et al. (2022) (arXiv:2209.14901, 2022)
What this evaluates
Evaluates generative models' ability to perform clinical diagnostic reasoning, including medical knowledge representation, evidence synthesis, and diagnosis generation. The benchmark spans sentence-level to full-note tasks to probe abstractive reasoning and clinical knowledge inference.
Datasets
- DR.BENCH — total ?; splits: train (-1), val (-1), test (-1)
Metrics
accuracy(primary) — range: percent- Proportion of correctly predicted labels or answers out of total instances.
macro F1— range: percent- Unweighted mean of recall or precision across all classes, typically used for relation labeling tasks.
ROUGE-L— range: percent- Longest common subsequence-based recall and precision score between generated summary and reference summary.
Input / output format
Input: Clinical text inputs such as progress notes, assessment sections, or medical board exam questions, sometimes with retrieved context paragraphs.
Output: Sequence generation: diagnostic labels, assessment-plan relations, problem list summaries, or multiple-choice answers.
Scoring recipe
def score(predictions, golds, metric):
if metric == 'accuracy':
return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
elif metric == 'macro F1':
return f1_score(golds, predictions, average='macro')
elif metric == 'ROUGE-L':
return rouge_score(golds, predictions, rouge_types=['rougeL'])
Common pitfalls
- Different tasks use different evaluation metrics (accuracy, F1, ROUGE-L), complicating direct cross-task comparison.
- 95% confidence intervals are computed via bootstrapping, not analytical formulas.
- Summarization tasks report very low ROUGE-L scores, which may not fully capture clinical reasoning quality.
Evidence (verbatim from paper)
The models achieved the best performance on MedNLI with an accuracy range between 79.75% and 84.88% (Table 4). Problem summarization (SUMM-NOTE), which was intended as the most challenging task, had the lowest performance across all models, with Rouge-L scores between 2.14% and 5.66% (Table 7).
Citation
@misc{gao2022drbench,
title={DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing},
author={Gao et al. (2022)},
year={2022},
note={arXiv:2209.14901}
}
- arXiv: 2209.14901