Fg Cxr Eval

fg-cxr-eval

FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation — Trong Thang Pham et al. (2024) (arXiv:2411.15413, 2024)

What this evaluates

Evaluates a model's ability to generate accurate, clinically correct chest X-ray reports while aligning its visual focus with radiologist gaze patterns. It probes both natural language generation quality and interpretable attention prediction to ensure diagnostic reasoning is visually grounded.

Datasets

• FG-CXR — total ?; splits: test (-1); repo https://github.com/UARK-AICV/FG-CXR

Metrics

• B4 — range: [0, 1]
  • BLEU-4 score measuring 4-gram overlap between generated and reference reports.
• C (primary) — range: [0, 1]
  • CIDEr score using TF-IDF weighted n-gram similarity to match generated reports with references.
• R — range: [0, 1]
  • ROUGE-L score measuring longest common subsequence overlap.
• M — range: [0, 1]
  • METEOR score incorporating synonymy and stemming for semantic matching.
• Div@2 — range: [0, 1]
  • Diversity metric measuring average pairwise similarity between generated reports to prevent repetitive single-sentence outputs.
• R@4 — range: [0, 1]
  • Recall metric evaluating coverage of reference content in generated reports.
• F1_ex (primary) — range: [0, 1]
  • Example-based F1 score aggregating precision and recall at the instance level to measure clinical correctness.
• fwIoU (primary) — range: percent
  • Frequency-weighted Intersection over Union between predicted and ground-truth gaze heatmaps.
• SSIM — range: [0, 1]
  • Structural Similarity Index measuring perceptual similarity between predicted and expert heatmaps.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio in decibels quantifying pixel-level reconstruction error between heatmaps.
• L1 — range: [0, 1]
  • Mean absolute error between predicted and ground-truth heatmap pixels.
• L2 — range: [0, 1]
  • Mean squared error between predicted and ground-truth heatmap pixels.
• fgIoU — range: percent
  • Intersection over Union computed over foreground (anatomical) regions in the heatmaps.
• bgIoU — range: percent
  • Intersection over Union computed over background regions in the heatmaps.

Input / output format

Input: Chest X-ray image

Output: Natural language radiology report and a predicted gaze attention heatmap

Scoring recipe

def evaluate(predictions, gold):
    # NLG metrics
    bleu4 = compute_bleu(predictions, gold, n=4)
    cider = compute_cider(predictions, gold)
    rouge_l = compute_rouge(predictions, gold, 'rougeL')
    meteor = compute_meteor(predictions, gold)
    div2 = compute_diversity(predictions, k=2)
    r4 = compute_recall(predictions, gold, k=4)
    # Clinical Efficacy
    f1_ex = compute_f1(predictions, gold, aggregation='example')
    # Attention Similarity
    fw_iou = compute_weighted_iou(pred_heatmap, gold_heatmap)
    ssim = compute_ssim(pred_heatmap, gold_heatmap)
    psnr = compute_psnr(pred_heatmap, gold_heatmap)
    l1 = compute_l1_loss(pred_heatmap, gold_heatmap)
    l2 = compute_l2_loss(pred_heatmap, gold_heatmap)
    return {'B4': bleu4, 'C': cider, 'R': rouge_l, 'M': meteor,
            'Div@2': div2, 'R@4': r4, 'F1_ex': f1_ex,
            'fwIoU': fw_iou, 'SSIM': ssim, 'PSNR': psnr,
            'L1': l1, 'L2': l2}

Common pitfalls

• Relying solely on NLG metrics (e.g., BLEU, ROUGE) for clinical correctness, as they are ill-suited for measuring medical accuracy.
• Ignoring diversity metrics (Div@2, R@4), which are required to detect models that generate repetitive single-sentence outputs.
• Evaluating attention using standard self-attention maps instead of aligning predictions with expert radiologist gaze heatmaps.

Evidence (verbatim from paper)

Follow [[34], [43], [41]], we evaluate our method based on three criteria: Natural Language Generation (NLG) metrics: BLEU (B), METEOR (M), ROUGE-L (R), and CIDEr (C) for matching generated report with the reference report; Div@2[[41]] and R@4[[52]] for diversity generated reports because we observe that report generation model can suffer heavily from generating only one sentence for all samples. • Clinical Efficacy (CE) metrics: we use all micro-, macro-, and example-based Precision, Recall, and F1 score described in [[13]] because NLG metrics alone are ill-suited for measuring clinical correctness [[43]]. • Attention Similarity metrics: we report all foreground IoU (fgIoU), background IoU (bgIoU), frequency-weighted IoU (fwIoU), Structural Similarity (SSIM), Peak signal-to-noise ratio (PSNR), L1, and L2.

Citation

@misc{pham2024fgcxr,
  title={FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation},
  author={Trong Thang Pham et al. (2024)},
  year={2024},
  note={arXiv:2411.15413}
}

• arXiv: 2411.15413