Chest Xray Report Generation Eval

chest-xray-report-generation-eval

Variational Topic Inference for Chest X-Ray Report Generation — Najdenkoska et al. (2021) (arXiv:2107.07314, 2021)

What this evaluates

This evaluation probes a model's ability to generate clinically coherent and accurate natural language reports from chest X-ray images. It measures both surface-level linguistic similarity to ground-truth radiology reports and the clinical correctness of extracted pathological findings.

Datasets

• Indiana U. Chest X-Ray — total 3195; splits: train (-1), val (-1), test (-1)
• MIMIC-CXR — total 218101; splits: train (-1), val (-1), test (-1)

Metrics

• BLEU-1 (primary) — range: [0, 1]
  • Computes unigram precision between generated and reference reports, penalizing overly short outputs via a brevity penalty. Scores range from 0 to 1.
• ROUGE — range: [0, 1]
  • Measures recall-based n-gram overlap (typically ROUGE-L) between generated and reference reports. Scores range from 0 to 1.
• METEOR — range: [0, 1]
  • Combines unigram precision and recall with penalties for synonymy, stemming, and word order mismatches. Scores range from 0 to 1.
• Clinical F1 — range: [0, 1]
  • Harmonic mean of precision and recall computed on binary pathological labels extracted from reports using the CheXpert rule-based labeler. Scores range from 0 to 1.

Input / output format

Input: Chest X-ray radiograph images normalized and resized to 224×224 pixels, processed through a pre-trained DenseNet-121 to extract visual features.

Output: A natural language chest X-ray report consisting of concatenated, lower-cased, and tokenized findings and impressions.

Scoring recipe

def evaluate(predictions, references):
    # NLG metrics
    bleu_1 = nltk.translate.bleu_score.sentence_bleu([ref.split() for ref in references], pred.split(), weights=(1,0,0,0))
    rouge = rouge_scorer.score(ref, pred)['rougeL'].fmeasure
    meteor = meteor_score.score(ref, pred)
    # Clinical metrics via CheXpert labeler
    pred_labels = chexpert_labeler(pred)
    gold_labels = chexpert_labeler(ref)
    tp = sum(p & g for p, g in zip(pred_labels, gold_labels))
    fp = sum(p & ~g for p, g in zip(pred_labels, gold_labels))
    fn = sum(~p & g for p, g in zip(pred_labels, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return bleu_1, rouge, meteor, precision, recall, f1

Common pitfalls

• Standard NLG metrics (BLEU/ROUGE) inherently penalize paraphrasing and diversity, which contradicts the model's goal of capturing radiological uncertainty and sentence-level variability.
• Clinical efficacy scores depend entirely on the CheXpert rule-based labeler, which may misclassify or miss subtle findings compared to expert radiologists.
• Generated reports are systematically longer than ground truth, which can artificially inflate recall-based metrics like ROUGE and F1.

Evidence (verbatim from paper)

We adopt commonly used evaluation metrics for natural language generation (NLG), including BLEU, METEOR and ROUGE. We compare to several other neural network based state-of-the-art methods: [[17], [19], [32], [33], [10], [3]] for Indiana U. X-Rays, and [[19], [3], [20]] for MIMIC-CXR. As shown in Table 1, our VTI achieves comparable performance or yields higher scores in terms of BLEU-1-2-3, ROUGE (for Indiana U. Chest X-ray) and METEOR (for MIMIC-CXR). As an additional evaluation in terms of the clinical coherence and correctness, we employ clinical efficacy metrics, i.e., precision, recall and F1 score [[19]] to compare the extracted labels by the rule-based CheXpert labeler [[9]] for the ground-truth and generated reports.

Citation

@misc{najdenkoska2021variational,
  title={Variational Topic Inference for Chest X-Ray Report Generation},
  author={Najdenkoska et al. (2021)},
  year={2021},
  note={arXiv:2107.07314}
}

• arXiv: 2107.07314