Mammography Report Eval

mammography-report-eval

MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation — Raiyan Jahangir et al. (2026) (arXiv:2602.22462, 2026)

What this evaluates

Evaluates the ability of local vision-language models to generate clinically styled mammography reports and perform multi-task classification (e.g., BI-RADS, breast density, calcifications) from medical images. It probes the models' robustness under zero-shot, few-shot, Chain-of-Thought prompting, and Retrieval-Augmented Generation (RAG), as well as the impact of parameter-efficient fine-tuning (QLoRA).

Datasets

• VinDr-Mammo — total ?; splits: test (-1)
• DMID — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, computed as macro-average across classes to handle class imbalance.
• BERTScore — range: [0, 1]
  • Similarity metric using contextual embeddings from BERT to compute precision, recall, and F1 between generated and reference text.
• ROUGE-L — range: [0, 1]
  • Recall-oriented overlap of the longest common subsequence between generated and reference text.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted labels out of total instances.

Input / output format

Input: Mammography images paired with prompts (zero-shot, few-shot, Chain-of-Thought, or RAG-augmented with retrieved clinical examples).

Output: Structured JSON containing fields for BI-RADS, Breast Density, Calcification, Mass, Asymmetry, Suspicion, and Findings text; or single-task generation of one field at a time.

Scoring recipe

def evaluate(predictions, gold):
    # Classification metrics
    acc = accuracy_score(gold['labels'], predictions['labels'])
    f1 = f1_score(gold['labels'], predictions['labels'], average='macro')
    # Text similarity metrics
    bert = bertscore.compute(predictions=predictions['text'], references=gold['text'])['f1']
    rouge = rouge_score(gold['text'], predictions['text'], rouge_types=['rougeL'])['rougeL']
    return {'accuracy': acc, 'f1': f1, 'bertscore': bert, 'rougeL': rouge}

Common pitfalls

• RAG can degrade performance on certain fields (e.g., findings text) if retrieved context mismatches the target distribution or introduces lexical noise.
• Fine-tuning output format (multi-task full JSON vs. single-task one-field) significantly impacts results; single-task may outperform multi-task for some labels but fail for others.
• Optimal training epochs are task-dependent and non-monotonic; early stopping or checkpoint selection is required rather than assuming more epochs always improve performance.

Evidence (verbatim from paper)

We first evaluate report-generation similarity using only text-similarity metrics (BERTScore and ROUGE-L) on narrative fields, BI-RADS, density, and findings directly read by clinicians.

Citation

@misc{jahangir2026mammowise,
  title={MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation},
  author={Raiyan Jahangir et al. (2026)},
  year={2026},
  note={arXiv:2602.22462}
}

• arXiv: 2602.22462