Medec Eval

medec-eval

A Systematic Analysis of Large Language Models with RAG-enabled Dynamic Prompting for Medical Error Detection and Correction — Ahmed et al. (2025) (arXiv:2511.19858, 2025)

What this evaluates

Evaluates large language models' ability to detect and correct medical errors in clinical text. It probes the model's sensitivity to clinical inaccuracies, its precision in localizing erroneous sentences, and its capacity to generate semantically and lexically accurate corrections using different prompting strategies.

Datasets

• MEDEC — total ?; splits: test (-1); repo https://github.com/abachaa/MEDEC

Metrics

• Error Flag Detection Accuracy — range: [0, 1]
  • Binary classification accuracy for identifying whether a clinical text contains at least one medical error.
• Error Sentence Detection Accuracy — range: [0, 1]
  • Accuracy of correctly identifying the specific sentence(s) within a text that contain(s) a medical error.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap between the model's corrected text and the ground-truth corrected text.
• BERTScore — range: [0, 1]
  • Contextual embedding-based similarity between predicted and reference corrections, computed via F1 score of token-level precision and recall.
• BLEURT — range: [0, 1]
  • Learned evaluation metric based on BERT that scores the semantic quality of the corrected text against the reference.
• AggScore (primary) — range: [0, 1]
  • Aggregate correction quality score, computed as the average of ROUGE-1, BERTScore, and BLEURT.
• False Positive Rate (FPR) — range: [0, 1]
  • Proportion of error-free sentences incorrectly flagged as erroneous by the model.

Input / output format

Input: Clinical text (medical notes) accompanied by task-specific instructions for error flagging, error sentence localization, and text correction. Retrieval-augmented variants also include dynamically retrieved clinical exemplars.

Output: Binary error flag (0/1), index or selection of the erroneous sentence, and the fully corrected clinical text.

Scoring recipe

def score(predictions, gold):
    flag_acc = mean(pred.flag == gold.flag)
    sent_acc = mean(pred.sentence_idx == gold.sentence_idx)
    rouge1 = rouge_1(pred.corrected_text, gold.corrected_text)
    bert = bertscore(pred.corrected_text, gold.corrected_text)
    bleu = bleurt(pred.corrected_text, gold.corrected_text)
    agg = (rouge1 + bert + bleu) / 3
    fpr = sum(pred.flag == 1 and gold.flag == 0) / sum(gold.flag == 0)
    return {'flag_acc': flag_acc, 'sent_acc': sent_acc, 'rouge1': rouge1, 'bert': bert, 'bleu': bleu, 'agg': agg, 'fpr': fpr}

Common pitfalls

• Confusing error flag detection (binary text-level) with error sentence detection (sentence-level localization), which have different accuracy baselines.
• Relying solely on ROUGE-1 for correction quality, which penalizes valid paraphrases and clinical shorthand substitutions that do not match the reference lexically.
• Ignoring False Positive Rate (FPR), as high detection accuracy can be achieved by over-flagging correct sentences, leading to unnecessary and harmful over-corrections.

Evidence (verbatim from paper)

Table 2 presents the results across all evaluation metrics. Compared to the zero-shot baseline, SPR one-shot improved both error flag and error sentence detection (from 0.6812 to 0.7016 and 0.6573 to 0.6670, respectively), along with corresponding gains in error correction. SPR ten-shot provided only modest additional improvements (error flag accuracy 0.7124, error sentence detection accuracy 0.6702), suggesting that exemplar count alone offers limited benefit. By contrast, RDP one-shot consistently outperformed its static counterpart (error flag detection accuracy 0.7168, error sentence detection accuracy 0.6810), demonstrating that exemplar quality matters even when exemplar count is held constant. Finally, RDP ten-shot achieved the strongest overall performance (GPT-4.1: error flag detection accuracy 0.7286, error sentence detection accuracy 0.7037, error correction AggScore 0.6707), confirming that exemplary quality and quantity are complementary.

Citation

@misc{ahmed2025medec,
  title={A Systematic Analysis of Large Language Models with RAG-enabled Dynamic Prompting for Medical Error Detection and Correction},
  author={Ahmed et al. (2025)},
  year={2025},
  note={arXiv:2511.19858}
}

• arXiv: 2511.19858