Cxpmrg Bench Eval

cxpmrg-bench-eval

CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset — Xiao Wang et al. (arXiv:2410.00379, 2024)

What this evaluates

This benchmark evaluates the capability of vision-language models to generate accurate and clinically relevant free-text radiology reports from chest X-ray images. It probes both linguistic quality through standard NLG metrics and diagnostic accuracy by extracting and comparing clinical abnormality labels against ground truth reports.

Datasets

• IU X-ray — total 7470; splits: train (-1), test (-1), val (-1)
• MIMIC-CXR — total 377110; splits: train (270790), val (2130), test (3858)
• CheXpert Plus — total 223228; splits: train (40463), val (5780), test (11562); repo https://github.com/Stanford-AIMI/chexpert-plus

Metrics

• CIDEr (primary) — range: [0, 1]
  • Evaluates text through TF-IDF weighted n-gram matching, placing greater emphasis on the importance of words. Computed as the mean score across all test samples.
• BLEU-4 — range: [0, 1]
  • Evaluates text quality through 4-gram matching with brevity penalty. Standard BLEU-4 formulation.
• ROUGE-L — range: [0, 1]
  • Evaluates using the longest common subsequence (LCS) between predicted and ground truth reports, measuring sequence-level recall and precision.
• METEOR — range: [0, 1]
  • Improves upon BLEU by considering synonyms, stemming, and word order alignment. Harmonic mean of precision and recall.
• CE-F1 — range: [0, 1]
  • Clinical Efficacy F1-score. Uses the CheXpert toolkit to extract binary labels for clinical abnormalities from both predicted and ground truth reports, then computes F1 = 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Chest X-ray image (DICOM or PNG format). During autoregressive decoding, the model also receives previously generated tokens as context.

Output: Free-text radiology report, typically structured into Findings and/or Impression sections.

Scoring recipe

def compute_metrics(predictions, golds):
    # NLG metrics
    bleu4 = compute_bleu(predictions, golds, n=4)
    rouge_l = compute_rouge_l(predictions, golds)
    meteor = compute_meteor(predictions, golds)
    cider = compute_cider(predictions, golds)  # TF-IDF weighted n-gram
    
    # Clinical Efficacy (CheXpert toolkit)
    pred_labels = chexpert_extract_labels(predictions)
    gold_labels = chexpert_extract_labels(golds)
    tp = sum(p in gold_labels for p in pred_labels)
    fp = sum(p not in gold_labels for p in pred_labels)
    fn = sum(g not in pred_labels for g in gold_labels)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return {'BLEU-4': bleu4, 'ROUGE-L': rouge_l, 'METEOR': meteor, 'CIDEr': cider, 'CE-F1': f1}

Common pitfalls

• Ground truth definition varies across datasets: IU X-ray and MIMIC-CXR baselines often use 'Findings' as ground truth, while some use 'Impression concatenated with Findings' (marked with † in tables). Mixing these causes unfair comparisons.
• Split protocols are dataset-specific and must be strictly followed (e.g., R2GenGPT protocol for IU/MIMIC, R2GenCSR protocol for CheXpert Plus) to match reported baselines.
• CE metrics require an external toolkit (CheXpert) for label extraction; differences in tokenization, synonym mapping, or label thresholds can significantly alter Precision/Recall/F1 scores.

Evidence (verbatim from paper)

For the X-ray medical report generation, we evaluate the model using widely used natural language generation (NLG) metrics, including CIDEr, BLEU, ROUGE-L, and METEOR. ... To measure the accuracy of descriptions for clinical abnormalities, we also report Clinical Efficacy (CE) metrics. CE metrics require the use of the CheXPert toolkit to first extract labels from predictive reports and ground truth, and then to compare the presence status of important clinical observations to capture the diagnostic accuracy of the generated reports. We use Precision, Recall, and F1 to evaluate model performance for clinical efficacy metrics.

Citation

@misc{wang2024cxpmrgbench,
  title={CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset},
  author={Xiao Wang et al.},
  year={2024},
  note={arXiv:2410.00379}
}

• arXiv: 2410.00379