# Fg Cxr Eval

> Evaluates a model's ability to generate accurate, clinically correct chest X-ray reports while aligning its visual focus with radiologist gaze patterns. It probes both natural language generation quality and interpretable attention prediction to ensure diagnostic reasoning is visually grounded. Use when the user wants to benchmark on FG-CXR, or asks about evaluating this task. Reports C, F1_ex, fwIoU.

- Skill: `qhjqhj00/fg-cxr-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/fg-cxr-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/fg-cxr-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/fg-cxr-eval

---


# fg-cxr-eval

> FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation — Trong Thang Pham et al. (2024) (arXiv:2411.15413, 2024)

## What this evaluates

Evaluates a model's ability to generate accurate, clinically correct chest X-ray reports while aligning its visual focus with radiologist gaze patterns. It probes both natural language generation quality and interpretable attention prediction to ensure diagnostic reasoning is visually grounded.

## Datasets

- **FG-CXR** — total ?; splits: test (-1); repo https://github.com/UARK-AICV/FG-CXR

## Metrics

- `B4` — range: [0, 1]
  - BLEU-4 score measuring 4-gram overlap between generated and reference reports.
- `C` **(primary)** — range: [0, 1]
  - CIDEr score using TF-IDF weighted n-gram similarity to match generated reports with references.
- `R` — range: [0, 1]
  - ROUGE-L score measuring longest common subsequence overlap.
- `M` — range: [0, 1]
  - METEOR score incorporating synonymy and stemming for semantic matching.
- `Div@2` — range: [0, 1]
  - Diversity metric measuring average pairwise similarity between generated reports to prevent repetitive single-sentence outputs.
- `R@4` — range: [0, 1]
  - Recall metric evaluating coverage of reference content in generated reports.
- `F1_ex` **(primary)** — range: [0, 1]
  - Example-based F1 score aggregating precision and recall at the instance level to measure clinical correctness.
- `fwIoU` **(primary)** — range: percent
  - Frequency-weighted Intersection over Union between predicted and ground-truth gaze heatmaps.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index measuring perceptual similarity between predicted and expert heatmaps.
- `PSNR` — range: other
  - Peak Signal-to-Noise Ratio in decibels quantifying pixel-level reconstruction error between heatmaps.
- `L1` — range: [0, 1]
  - Mean absolute error between predicted and ground-truth heatmap pixels.
- `L2` — range: [0, 1]
  - Mean squared error between predicted and ground-truth heatmap pixels.
- `fgIoU` — range: percent
  - Intersection over Union computed over foreground (anatomical) regions in the heatmaps.
- `bgIoU` — range: percent
  - Intersection over Union computed over background regions in the heatmaps.

## Input / output format

**Input**: Chest X-ray image

**Output**: Natural language radiology report and a predicted gaze attention heatmap

## Scoring recipe

```python
def evaluate(predictions, gold):
    # NLG metrics
    bleu4 = compute_bleu(predictions, gold, n=4)
    cider = compute_cider(predictions, gold)
    rouge_l = compute_rouge(predictions, gold, 'rougeL')
    meteor = compute_meteor(predictions, gold)
    div2 = compute_diversity(predictions, k=2)
    r4 = compute_recall(predictions, gold, k=4)
    # Clinical Efficacy
    f1_ex = compute_f1(predictions, gold, aggregation='example')
    # Attention Similarity
    fw_iou = compute_weighted_iou(pred_heatmap, gold_heatmap)
    ssim = compute_ssim(pred_heatmap, gold_heatmap)
    psnr = compute_psnr(pred_heatmap, gold_heatmap)
    l1 = compute_l1_loss(pred_heatmap, gold_heatmap)
    l2 = compute_l2_loss(pred_heatmap, gold_heatmap)
    return {'B4': bleu4, 'C': cider, 'R': rouge_l, 'M': meteor,
            'Div@2': div2, 'R@4': r4, 'F1_ex': f1_ex,
            'fwIoU': fw_iou, 'SSIM': ssim, 'PSNR': psnr,
            'L1': l1, 'L2': l2}
```

## Common pitfalls

- Relying solely on NLG metrics (e.g., BLEU, ROUGE) for clinical correctness, as they are ill-suited for measuring medical accuracy.
- Ignoring diversity metrics (Div@2, R@4), which are required to detect models that generate repetitive single-sentence outputs.
- Evaluating attention using standard self-attention maps instead of aligning predictions with expert radiologist gaze heatmaps.

## Evidence (verbatim from paper)

> Follow [[34], [43], [41]], we evaluate our method based on three criteria: Natural Language Generation (NLG) metrics: BLEU (B), METEOR (M), ROUGE-L (R), and CIDEr (C) for matching generated report with the reference report; Div@2[[41]] and R@4[[52]] for diversity generated reports because we observe that report generation model can suffer heavily from generating only one sentence for all samples. • Clinical Efficacy (CE) metrics: we use all micro-, macro-, and example-based Precision, Recall, and F1 score described in [[13]] because NLG metrics alone are ill-suited for measuring clinical correctness [[43]]. • Attention Similarity metrics: we report all foreground IoU (fgIoU), background IoU (bgIoU), frequency-weighted IoU (fwIoU), Structural Similarity (SSIM), Peak signal-to-noise ratio (PSNR), L1, and L2.

## Citation

```bibtex
@misc{pham2024fgcxr,
  title={FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation},
  author={Trong Thang Pham et al. (2024)},
  year={2024},
  note={arXiv:2411.15413}
}
```

- arXiv: 2411.15413