Iu Xray Report Gen Eval

iu-xray-report-gen-eval

LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning — Md. Zihad Bin Jahangir et al. (2025) (arXiv:2506.03178, 2025)

What this evaluates

Evaluates a vision-language model's ability to generate clinically accurate and semantically coherent radiology reports from chest X-ray images. It probes the model's capacity for medical terminology usage, anatomical consistency, and structured clinical text generation.

Datasets

• IU X-ray — total ?; splits: test (-1)

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Measures the longest common subsequence (LCS) between generated and reference reports to assess structural and semantic fidelity.
• METEOR — range: [0, 1]
  • Computes alignment between generated and reference text using synonymy, stemming, and semantic matching to evaluate linguistic fluency and semantic accuracy.

Input / output format

Input: Chest X-ray image (optionally with attention map visualization)

Output: Natural language radiology report describing findings, impressions, and anatomical observations.

Scoring recipe

def compute_metrics(predictions, references):
    rouge_l_scores = [rouge_scorer.score(ref, pred)['rougeL'].fmeasure for pred, ref in zip(predictions, references)]
    meteor_scores = [meteor_score([ref], pred) for pred, ref in zip(predictions, references)]
    return {
        'ROUGE-L': sum(rouge_l_scores) / len(predictions),
        'METEOR': sum(meteor_scores) / len(predictions)
    }

Common pitfalls

• Table 3 results for baseline methods are quoted from their original publications, not re-evaluated on the same split.
• BLEU metrics heavily penalize synonymy and paraphrasing, making them less reliable for clinical text than ROUGE-L or METEOR.
• The dataset split sizes and exact preprocessing steps for IU X-ray are not detailed in this paper.

Evidence (verbatim from paper)

LLaMA-XR demonstrates superior performance on ROUGE-L and METEOR, two widely used metrics that emphasize semantic fidelity and syntactic fluency. In particular, LLaMA-XR achieves a 4.34% improvement in ROUGE-L over the state-of-the-art best result from the SGF method.

Citation

@misc{jahangir2025llamaxr,
  title={LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning},
  author={Md. Zihad Bin Jahangir et al. (2025)},
  year={2025},
  note={arXiv:2506.03178}
}

• arXiv: 2506.03178