Dmid Mammography Report Eval

dmid-mammography-report-eval

AMRG: Extend Vision Language Models for Automatic Mammography Report Generation — Sung et al. (2025) (arXiv:2508.09225, 2025)

What this evaluates

Evaluates a vision-language model's ability to generate clinically accurate and linguistically fluent mammography reports from multi-view breast images. It probes both natural language generation quality and domain-specific diagnostic reasoning, specifically BI-RADS categorization and breast density assessment.

Datasets

• DMID — total ?; splits: train (-1), test (-1)

Metrics

• ROUGE-L — range: [0, 1]
  • Measures the overlap of longest common subsequences between generated and reference reports, weighted by recall.
• METEOR — range: [0, 1]
  • Computes a weighted harmonic mean of precision and recall, incorporating synonymy and stemming to capture semantic similarity.
• CIDEr — range: [0, 1]
  • Ranks n-grams by inverse document frequency to penalize common words and reward distinctive, clinically informative terms.
• BI-RADS Accuracy (primary) — range: [0, 1]
  • Proportion of exact matches between predicted and ground-truth BI-RADS diagnostic categories.
• Density Accuracy — range: [0, 1]
  • Proportion of exact matches between predicted and ground-truth breast density labels.

Input / output format

Input: Multi-view mammography images.

Output: Textual mammography report containing descriptive findings, BI-RADS category, and breast density assessment.

Scoring recipe

def compute_metrics(predictions, golds):
    nlp_scores = {}
    for metric in ['bleu-1','rouge-1','rouge-2','rouge-l','meteor','cider','f1']:
        nlp_scores[metric] = evaluate_nlp(predictions, golds, metric)
    birads_pred = extract_label(predictions, 'birads')
    birads_gold = extract_label(golds, 'birads')
    density_pred = extract_label(predictions, 'density')
    density_gold = extract_label(golds, 'density')
    nlp_scores['BI-RADS Accuracy'] = sum(p==g for p,g in zip(birads_pred, birads_gold)) / len(golds)
    nlp_scores['Density Accuracy'] = sum(p==g for p,g in zip(density_pred, density_gold)) / len(golds)
    return nlp_scores

Common pitfalls

• Using high LoRA ranks (e.g., r=64) causes overfitting on the relatively small DMID dataset, degrading both NLP and clinical metrics.
• Surface-level n-gram metrics (BLEU-1, ROUGE-2) may favor syntactically fluent but clinically inaccurate outputs, whereas semantic metrics and exact label matching better reflect diagnostic utility.
• General-purpose VLMs often hallucinate benign structures or omit critical findings despite high fluency scores.

Evidence (verbatim from paper)

In particular, the configuration $(r=32,\alpha=16)$ achieves the highest scores across all NLP metrics (e.g., ROUGE-L 0.52, METEOR 0.5194, CIDEr 0.5336) and clinical metrics (BI-RADS accuracy 0.55, density accuracy 0.35), outperforming both the base model and other LoRA variants.

Citation

@misc{sung2025amrg,
  title={AMRG: Extend Vision Language Models for Automatic Mammography Report Generation},
  author={Sung et al. (2025)},
  year={2025},
  note={arXiv:2508.09225}
}

• arXiv: 2508.09225