Vimed Pet Eval

vimed-pet-eval

Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation — Nguyen et al. (2025) (arXiv:2509.24739, 2025)

What this evaluates

Evaluates vision-language models on generating Vietnamese clinical reports from paired PET/CT images and answering medical questions about them. Probes the model's ability to align 3D medical imaging features with low-resource language text and produce clinically accurate descriptions.

Datasets

• ViMed-PET — total ?; splits: test (-1)

Metrics

• BLEU-4 (primary) — range: [0, 1]
  • 4-gram precision with a brevity penalty to penalize overly short or fragmented generations.
• ROUGE-1 — range: [0, 1]
  • Unigram recall/precision measuring word-level overlap between generated and reference text.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence score capturing sentence-level fluency and structural alignment.
• BERT score — range: [0, 1]
  • Contextual embedding similarity (F1) between generated and reference texts using a pretrained BERT model.
• F1-T — range: [0, 1]
  • Clinical entity-level F1 score for general medical terms.
• F1-TP — range: [0, 1]
  • Clinical entity-level F1 score for tumor-related terms.
• F1-TF — range: [0, 1]
  • Clinical entity-level F1 score for treatment-related terms.
• F1-TPF — range: [0, 1]
  • Clinical entity-level F1 score for combined tumor, treatment, and follow-up terms.

Input / output format

Input: Paired 3D PET/CT images, and for the VQA task, a natural language question about the scan.

Output: A Vietnamese clinical report (for report generation) or a short text answer (for VQA).

Scoring recipe

def compute_metrics(predictions, references):
    bleu4 = compute_bleu(predictions, references, n=4)
    rouge1 = compute_rouge(predictions, references, n=1)
    rouge_l = compute_rouge(predictions, references, n='l')
    bert = compute_bert_score(predictions, references)
    # Clinical F1 metrics require entity-level matching specific to Vietnamese medical terminology
    f1_t, f1_tp, f1_tf, f1_tpf = compute_clinical_f1(predictions, references)
    return {'BLEU-4': bleu4, 'ROUGE-1': rouge1, 'ROUGE-L': rouge_l, 'BERT score': bert,
            'F1-T': f1_t, 'F1-TP': f1_tp, 'F1-TF': f1_tf, 'F1-TPF': f1_tpf}

Common pitfalls

• Metrics are reported as absolute scores, but the text heavily emphasizes relative percentage gains over baselines, which can cause confusion if not normalized correctly.
• Clinical F1 metrics (F1-T, F1-TP, etc.) are not standard NLP metrics and likely require entity-level matching specific to Vietnamese medical terminology, which is not detailed in the text.
• GPT-4o is evaluated via few-shot prompting while other models are fine-tuned, making direct comparison potentially unfair.

Evidence (verbatim from paper)

For instance, when LLaMA-2-7B is paired with either CT-ViT or our customized Cosmos Tokenizer and fine-tuned on ViMed-PET, it significantly outperforms the pretrained LLaMA-2-7B model used in M3D across all key metrics (i.e., BLEU-4, ROUGE-1, ROUGE-L, and BERT score).

Citation

@misc{nguyen2025vimedpet,
  title={Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2509.24739}
}

• arXiv: 2509.24739