C Srrg Eval

c-srrg-eval

Automated Structured Radiology Report Generation with Rich Clinical Context — Kang et al. (2025) (arXiv:2510.00428, 2025)

What this evaluates

Evaluates multimodal large language models on automated structured radiology report generation, specifically testing their ability to produce clinically accurate findings and impressions while integrating rich clinical context (multi-view X-rays, indications, techniques, prior studies) to mitigate temporal hallucinations.

Datasets

• C-SRRG — total ?; splits: Valid (-1), Test (-1), Test-reviewed (-1); repo https://github.com/vuno/contextualized-srrg

Metrics

• BLEU — range: percent
  • N-gram precision score measuring exact word overlap between generated and reference reports. Computed using standard smoothing for short texts.
• ROUGE-L — range: percent
  • Recall-oriented metric based on the longest common subsequence between generated and reference text, capturing sentence-level fluency and structure.
• BERTScore — range: [0, 1]
  • Computes token-level cosine similarity between contextual embeddings of the generated and reference texts, capturing semantic similarity beyond exact n-gram matches.
• F1-RadGraph — range: [0, 1]
  • F1-score of clinical entity and relation extraction using the RadGraph pipeline, measuring clinical accuracy of extracted findings and observations.
• F1-SRRG-BERT (primary) — range: [0, 1]
  • F1-score computed using CXR-BERT embeddings for structured evaluation of radiology reports, aligning generated text with clinical ontology representations.
• Category Score — range: [0, 1]
  • F1-score for the correctness of organ-section headers (e.g., lungs, cardiovascular, pleura) in the generated findings, evaluated only for the findings task.
• Temporal Hallucination Rate — range: percent
  • Percentage of generated reports containing inappropriate temporal references (e.g., 'new', 'stable', 'improved') when evaluated without supporting clinical context.

Input / output format

Input: Chest X-ray images (single or multi-view) optionally accompanied by clinical context text (indication, technique, prior study comparisons).

Output: Structured radiology report text containing 'Findings' and 'Impression' sections.

Scoring recipe

def evaluate(predictions, golds):
    results = {}
    for pred, gold in zip(predictions, golds):
        results['BLEU'] = compute_bleu(pred, gold)
        results['ROUGE-L'] = compute_rouge_l(pred, gold)
        results['BERTScore'] = compute_bertscore(pred, gold)
        results['F1-RadGraph'] = compute_f1_radgraph(pred, gold)
        results['F1-SRRG-BERT'] = compute_f1_srrg_bert(pred, gold)
        results['Category Score'] = compute_category_score(pred, gold)
    return {k: np.mean([r[k] for r in results]) for k in results}

Common pitfalls

• Greedy decoding is enforced for reproducibility but may lower BLEU scores compared to sampling, especially for smaller models.
• Clinical context availability is heterogeneous; models trained without context may hallucinate temporally when context is provided at inference, and vice versa.
• Performance gains from context are scale-dependent; smaller models (3B) may show fluency degradation despite clinical accuracy improvements.

Evidence (verbatim from paper)

We use standard metrics, such as BLEU (Papineni et al., [2002]), ROUGE-L (Lin, [2004]), and BERTScore (Zhang et al., [2019]), to assess text quality. For clinical accuracy, we report F1-RadGraph (Delbrouck et al., [2022]) and SRRG-specific metrics(Delbrouck et al., [2025]): F1-SRRG-BERT, built on CXR-BERT(Boecking et al., [2022]) for structured evaluation, and Category Score (only for findings) for the correctness of organ-section headers.

Citation

@misc{kang2025contextualized,
  title={Automated Structured Radiology Report Generation with Rich Clinical Context},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2510.00428}
}

• arXiv: 2510.00428