Xray Report Gen Eval

xray-report-gen-eval

EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation — Zhang et al. (2025) (arXiv:2510.16776, 2025)

What this evaluates

Evaluates the capability of vision-language models to generate accurate and clinically relevant radiology reports from chest X-ray images. It probes both linguistic fluency and coverage against reference reports, as well as clinical accuracy in identifying pathological findings.

Datasets

• IU X-ray — total 7470; splits: train (-1), test (-1), val (-1)
• MIMIC-CXR — total 227835; splits: train (270790), val (2130), test (3858)
• CheXpert Plus — total 187711; splits: train (40463), val (5780), test (11562)

Metrics

• BLEU-1 — range: [0, 1]
  • 1-gram overlap between generated and reference reports, averaged over the test set.
• BLEU-2 — range: [0, 1]
  • 2-gram overlap between generated and reference reports, averaged over the test set.
• BLEU-3 — range: [0, 1]
  • 3-gram overlap between generated and reference reports, averaged over the test set.
• BLEU-4 — range: [0, 1]
  • 4-gram overlap between generated and reference reports, averaged over the test set.
• ROUGE-L (primary) — range: [0, 1]
  • Longest common subsequence recall and precision between generated and reference reports, averaged over the test set.
• METEOR — range: [0, 1]
  • Harmonic mean of precision and recall with synonym/stem matching, averaged over the test set.
• CIDEr — range: [0, 1]
  • TF-IDF weighted n-gram similarity between generated and reference reports, averaged over the test set.
• Precision — range: [0, 1]
  • Clinical metric: TP/(TP+FP), where TP/FP are correctly/incorrectly identified pathological findings.
• Recall — range: [0, 1]
  • Clinical metric: TP/(TP+FN), where TP/FN are correctly identified/missed pathological findings.
• F1 — range: [0, 1]
  • Clinical metric: 2×P×R/(P+R), harmonic mean of Precision and Recall for pathological findings.

Input / output format

Input: Chest X-ray images (frontal and/or lateral views) paired with patient demographic/contextual metadata.

Output: Free-text radiology report, typically structured into sections such as Findings, Impression, Indication, and Comparison.

Scoring recipe

def score(predictions, references):
    nlg = {'BLEU-4': 0, 'ROUGE-L': 0, 'METEOR': 0, 'CIDEr': 0}
    tp, fp, fn = 0, 0, 0
    for pred, ref in zip(predictions, references):
        for m in ['BLEU-4', 'ROUGE-L', 'METEOR', 'CIDEr']:
            nlg[m] += compute_metric(m, pred, ref)
        pred_set, ref_set = extract_diseases(pred), extract_diseases(ref)
        tp += len(pred_set & ref_set)
        fp += len(pred_set - ref_set)
        fn += len(ref_set - pred_set)
    n = len(predictions)
    nlg = {k: v/n for k, v in nlg.items()}
    p = tp/(tp+fp) if (tp+fp)>0 else 0
    r = tp/(tp+fn) if (tp+fn)>0 else 0
    f1 = 2*p*r/(p+r) if (p+r)>0 else 0
    return {**nlg, 'Precision': p, 'Recall': r, 'F1': f1}

Common pitfalls

• Ground truth selection varies across baselines; this protocol uses the 'Findings' section, whereas some competitors use 'Impression' or concatenated sections, making direct score comparison invalid without alignment.
• Dataset splits are not standardized; this work follows R2GenGPT/R2GenCSR partition protocols rather than official dataset splits, requiring exact match of train/val/test indices for fair comparison.
• Clinical metrics (Precision/Recall/F1) depend on the specific disease annotation scheme (e.g., RadGraph vs. CheXpert labels), which is not explicitly detailed in the metric definition section.

Evidence (verbatim from paper)

For evaluation metrics, we adopt natural language metrics and clinical metrics to evaluate our generated X-ray reports. For the natural language metrics, we choose BLEU, ROUGE-L, METEOR, and CIDEr. For the clinical metrics, i.e., Precision, Recall, and F1-measure, the formula can be formally defined as: Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2×P×R/(P+R) where TP (True Positive) refers to instances that are correctly identified as positive, FP (False Positive) denotes cases incorrectly labeled as positive when they are actually negative (also known as a Type I error), and FN (False Negative) represents instances incorrectly classified as negative despite being positive (referred to as a Type II error).

Citation

@misc{zhang2025emrrg,
  title={EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.16776}
}

• arXiv: 2510.16776