Lrrg Eval

lrrg-eval

Radiology Report Generation for Low-Quality X-Ray Images — Zhu et al. (2026) (arXiv:2604.10188, 2026)

What this evaluates

Evaluates a model's ability to generate accurate radiology reports from single-view X-ray images under varying quality conditions, from standard to severely degraded. It probes robustness to clinical acquisition artifacts and tests whether the model can extract quality-invariant diagnostic features without relying on historical patient data.

Datasets

• MIMIC-CXR LRRG Benchmarks — total 3478; splits: T_std (1780), T_mild (515), T_severe (514), T_aux (669)

Metrics

• BLEU-1 — range: [0, 1]
  • Unigram overlap between generated report and ground truth reference.
• BLEU-4 — range: [0, 1]
  • 4-gram overlap with brevity penalty between generated report and ground truth reference.
• METEOR — range: [0, 1]
  • Harmonic mean of unigram precision and recall with synonym/stem matching and penalty for fragmentation.
• ROUGE-L — range: [0, 1]
  • F1-score based on the longest common subsequence between generated and reference reports.
• CheXbert F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score across 14 standard pathology labels extracted via the CheXbert classifier, measuring diagnostic label alignment.
• RaTEScore — range: [0, 1]
  • Assesses entity-attribute alignment in generated reports using medical encoders to capture semantic consistency.
• RadGraph — range: [0, 1]
  • Evaluates preservation of complex clinical relationships by constructing and comparing knowledge graphs of entities and relations.

Input / output format

Input: Single-view frontal X-ray image (AP/PA). No historical patient data or multi-view scans are provided.

Output: Natural language radiology report describing findings, impressions, and clinical observations.

Scoring recipe

def score(predictions, references):
    # NLG metrics
    bleu1 = compute_bleu(predictions, references, n=1)
    bleu4 = compute_bleu(predictions, references, n=4)
    meteor = compute_meteor(predictions, references)
    rouge_l = compute_rouge(predictions, references, 'rougeL')
    
    # Clinical metrics via RadEval/CheXbert
    pred_labels = chexbert_labeler(predictions) # 14 pathology flags
    gold_labels = chexbert_labeler(references)
    precision, recall, f1 = precision_recall_f1(pred_labels, gold_labels)
    
    # Advanced semantic metrics
    rat_score = compute_ratescore(predictions, references)
    radgraph_score = compute_radgraph(predictions, references)
    
    return {
        'BLEU-1': bleu1, 'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l,
        'CheXbert_P': precision, 'CheXbert_R': recall, 'CheXbert_F1': f1,
        'RaTEScore': rat_score, 'RadGraph': radgraph_score
    }

Common pitfalls

• Relying solely on standard NLG metrics (BLEU/ROUGE) which correlate poorly with clinical correctness; the protocol explicitly prioritizes CheXbert F1, RaTEScore, and RadGraph for diagnostic evaluation.
• Providing multi-view images or historical exam data, which violates the strict single-view, no-history constraint designed to isolate image quality as the sole variable.
• Using complex, module-heavy baseline architectures that introduce confounding architectural priors; the evaluation mandates minimalist baselines (R2GenGPT, Qwen3-VL) to attribute performance shifts strictly to data quality.

Evidence (verbatim from paper)

For Natural Language Generation (NLG), we employ standard metrics including BLEU-1, BLEU-4, METEOR, and ROUGE-L. These metrics quantify the lexical fluency and n-gram overlap between the generated hypotheses and ground-truth references. While effective for measuring textual coherence, these standard metrics often fail to capture the precise medical correctness required for diagnostic reporting, necessitating a more specialized clinical evaluation. Consequently, we prioritize Clinical Efficacy (CE) by leveraging the RadEval toolkit... calculating Precision, Recall, and F1 scores based on diagnostic label alignment. To capture deeper semantic consistency beyond simple classification, we incorporate advanced metrics supported by RadEval: RaTEScore assesses entity-attribute alignment using medical encoders, RadGraph evaluates the preservation of complex clinical relationships through knowledge graph construction.

Citation

@misc{zhu2026radiology,
  title={Radiology Report Generation for Low-Quality X-Ray Images},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2604.10188}
}

• arXiv: 2604.10188