Iu Rr Radiology Report Eval

iu-rr-radiology-report-eval

Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment — Jianbo Yuan et al. (2019) (arXiv:1907.09085, 2019)

What this evaluates

Evaluates a model's ability to generate clinically accurate and structurally coherent radiology reports from multi-view chest X-ray images. It probes cross-modal alignment, medical terminology recall, and the model's capacity to synthesize findings and impressions from visual evidence.

Datasets

• IU-RR — total 3074; splits: train (2459), test (615)

Metrics

• BLEU-4 (primary) — range: [0, 1]
  • Computes the geometric mean of 1-gram to 4-gram precisions with a brevity penalty to penalize overly short translations. Standard n-gram overlap metric for text generation.
• METEOR — range: [0, 1]
  • Measures alignment between generated and reference text using exact, stem, synonym, and paraphrase matches, weighted more heavily on recall than precision.
• ROUGE — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation; computes n-gram recall between generated and reference texts to assess content coverage.

Input / output format

Input: Paired frontal and lateral chest X-ray images.

Output: Tokenized radiology report text consisting of concatenated 'findings' and 'impression' sections, with '' and '' tags at the beginning and end of each sentence.

Scoring recipe

def compute_bleu4(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        # ref is a list of lists (sentence-level tokens)
        score = sentence_bleu([ref], pred, weights=(0.25, 0.25, 0.25, 0.25))
        scores.append(score)
    return sum(scores) / len(scores)
# Note: predictions and references must follow the paper's preprocessing:
# - Concatenated findings + impression
# - <start>/<end> sentence tags
# - Low-freq words replaced with <unk>

Common pitfalls

• Failing to concatenate the 'findings' and 'impression' sections before evaluation, as the paper explicitly merges them for training/testing.
• Using the standard IU-RR splits instead of the paper's strict 80/20 split on the filtered 3,074 samples.
• Ignoring the ''/'' sentence boundary tokens and low-frequency word filtering ('') applied during preprocessing, which affects token alignment and metric calculation.

Evidence (verbatim from paper)

Since neither of the aforementioned datasets released radiology reports, we use IU-RR[[1]] for evaluating radiology report generation. For preprocessing, we first removed samples without multi-view images, and concatenated the “findings” and “impression” sections because in some forms all contents are either in the “findings” or “impression” section with the other left blank. We filtered out the reports with less than 3 sentences. In the end, we obtained 3,074 samples with multi-view images of which 20% (615 samples/1,330 images) are used for testing, and the 80% (2459 samples/4,918 images) are used for training and validation. The evaluation metrics we use are BLEU[[9]], METEOR[[2]], and ROUGE[[8]] scores, all of which are widely used in image captioning and machine translation tasks.

Citation

@misc{yuan2019automatic,
  title={Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment},
  author={Jianbo Yuan et al. (2019)},
  year={2019},
  note={arXiv:1907.09085}
}

• arXiv: 1907.09085