Radgpt Tumor Report Eval

radgpt-tumor-report-eval

RadGPT: Constructing 3D Image-Text Tumor Datasets — Bassi et al. (2025) (arXiv:2501.04678, 2025)

What this evaluates

Evaluates an AI pipeline's ability to generate clinically accurate radiology reports from 3D CT scans by detecting tumors, measuring their size, localizing them within organ sub-segments, and staging cancers. It also assesses the textual similarity and diagnostic utility of generated reports compared to ground-truth clinical notes.

Datasets

• AbdomenAtlas 3.0 — total ?; splits: test (-1); repo https://github.com/MrGiovanni/RadGPT

Metrics

• Tumor Detection Sensitivity & Specificity (primary) — range: [0, 1]
  • Calculated via a zero-shot LLM (Llama-3.1) that classifies whether a generated report indicates tumor presence. Sensitivity = TP / (TP + FN); Specificity = TN / (TN + FP). Validated against radiologist labels with 96% LLM accuracy.
• Tumor Size Measurement Accuracy — range: percent
  • Manual evaluation by a radiologist. A reported measurement is correct if it deviates by ≤10% from the radiologist's ground-truth measurement using WHO standards.
• Cancer Staging Accuracy — range: percent
  • Percentage of correctly predicted T stages (T1–T3) for pancreatic adenocarcinoma on a private dataset (N=42).
• Text Similarity (BLEU, METEOR, ROUGE-1/2/L, BERTScore) — range: [0, 100]
  • Standard n-gram and embedding-based overlap metrics comparing generated reports to ground-truth clinical notes. BLEU, METEOR, ROUGE, and BERTScore are computed per standard implementations.

Input / output format

Input: 3D CT scan volumes

Output: Radiology reports (structured or narrative) containing tumor presence, size, location, attenuation, volume, and cancer staging (e.g., T1–T4)

Scoring recipe

```python
# LLM-based detection sensitivity/specificity
reports = model.generate(ct_scans)
llm_preds = [llm.predict_tumor_present(r) for r in reports]
sensitivity = sum(p == 1 for p, g in zip(llm_preds, gt_labels) if g == 1) / sum(gt_labels)
specificity = sum(p == 0 for p, g in zip(llm_preds, gt_labels) if g == 0) / sum(1 - g for g in gt_labels)

# Manual size accuracy (subset)
correct = sum(1 for a, g in zip(ai_sizes, gt_sizes) if abs(a - g) / g <= 0.10)
size_accuracy = correct / len(ai_sizes)

## Common pitfalls

- Style variations (structured vs. narrative) heavily impact ROUGE and METEOR scores despite identical diagnostic content.
- Automatically matching AI-generated tumor measurements to ground-truth reports is difficult when multiple tumors are mentioned in the same sub-segment.
- LLM-based evaluation must be validated against radiologists, as zero-shot LLMs can hallucinate or misinterpret clinical phrasing without careful prompting.

## Evidence (verbatim from paper)

> RadGPT achieved 75.6% tumor detection precision and 93.8% tumor size measurement accuracy, according to manual evaluation. A radiologist evaluated the reports RadGPT created for 23 CT scans from UCSF. A reported tumor measurement was considered correct if it deviated by 10% or less from the radiologist's measurement (both use the WHO measuring standard [47]).

## Citation

```bibtex
@misc{bassi2025radgpt,
  title={RadGPT: Constructing 3D Image-Text Tumor Datasets},
  author={Bassi et al. (2025)},
  year={2025},
  note={arXiv:2501.04678}
}

• arXiv: 2501.04678