Medrect Eval

medrect-eval

MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts — Iwase et al. (2025) (arXiv:2511.00421, 2025)

What this evaluates

Evaluates large language models' ability to detect, localize, and correct clinical errors in medical texts across Japanese and English. It probes cross-lingual medical reasoning, precise error identification, and accurate text correction capabilities.

Datasets

• MedRECT-ja — total ?; splits: test (-1)
• MedRECT-en — total ?; splits: test (-1)

Metrics

• Error Detection F1 (primary) — range: [0, 1]
  • F1 score for binary classification determining whether a clinical error is present in the text.
• Sentence Extraction Accuracy — range: [0, 1]
  • Accuracy of predicting the correct sentence number containing the error (multi-class classification).
• ROUGE-1 F-score — range: [0, 1]
  • ROUGE-1 F1 computed with MeCab tokenizer for Japanese and whitespace tokenizer for English.
• BERTScore F1 — range: [0, 1]
  • BERTScore F1 using microsoft/deberta-xlarge-mnli as the base model with language-specific settings.
• BLEURT — range: [0, 1]
  • BLEURT score computed using the BLEURT-20 checkpoint.
• Error Correction Average — range: [0, 1]
  • Arithmetic average of ROUGE-1, BERTScore, and BLEURT scores.

Input / output format

Input: Zero-shot prompt providing a clinical text snippet containing an error, instructing the model to identify the error, specify the sentence number, and provide the corrected text.

Output: Structured response containing: (1) binary error detection, (2) sentence number localization, (3) corrected clinical text.

Scoring recipe

for each sample in test_set:
  if gold_has_error:
    det_f1 = compute_f1(pred_detect, gold_detect)
    loc_acc = compute_accuracy(pred_sentence_num, gold_sentence_num)
    if pred_detect == 'error':
      rouge1 = compute_rouge1(pred_correction, gold_correction, tokenizer='mecab' if lang=='ja' else 'whitespace')
      bertscore = compute_bertscore(pred_correction, gold_correction, model='deberta-xlarge-mnli')
      bleurt = compute_bleurt(pred_correction, gold_correction, model='bleurt-20')
      corr_avg = (rouge1 + bertscore + bleurt) / 3
  else:
    corr_avg = None
return det_f1, loc_acc, corr_avg

Common pitfalls

• Correction metrics (ROUGE-1, BERTScore, BLEURT) are only computed on samples where both the model prediction and ground truth indicate an error exists; ignoring this filtering inflates scores.
• Sentence extraction accuracy is only evaluated on samples with a ground-truth error, not on the full dataset.
• Japanese ROUGE-1 requires MeCab tokenization, while English uses whitespace tokenization; using a single tokenizer for both will yield incorrect scores.

Evidence (verbatim from paper)

We employed the following evaluation metrics: Error Detection F1 (binary classification), Sentence Extraction Accuracy (multi-class classification of sentence number), and Error Correction using ROUGE-1, BERTScore, BLEURT, and their arithmetic average. Following the MEDIQA-CORR 2024 evaluation protocol, sentence extraction is computed only on samples with a ground-truth error, and error correction metrics are computed only on samples where both prediction and ground-truth indicate the presence of an error.

Citation

@misc{iwase2025medrect,
  title={MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts},
  author={Iwase et al. (2025)},
  year={2025},
  note={arXiv:2511.00421}
}

• arXiv: 2511.00421