Medical Multimodal Eval

medical-multimodal-eval

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining — Bingqian Lin et al. (arXiv:2304.14204, 2023)

What this evaluates

Evaluates cross-modal understanding and generation capabilities in medical imaging, specifically image-report retrieval, radiology report generation, and multi-label disease diagnosis from chest X-rays.

Datasets

• MIMIC-CXR — total ?; splits: test (-1)
• IU-Xray — total ?; splits: test (-1)
• ChestX-ray 14 — total ?; splits: test (-1)

Metrics

• R@K — range: [0, 1]
  • Recall at K: the fraction of ground-truth items successfully retrieved within the top K predictions for both image-to-report and report-to-image retrieval.
• CIDEr (primary) — range: [0, 1]
  • TF-IDF weighted n-gram similarity between the generated report and reference reports, emphasizing rare but informative words.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve computed per disease class and averaged across all 14 categories.

Input / output format

Input: Per task: (1) IRR: chest X-ray image or radiology report; (2) MRG: chest X-ray image; (3) Diagnosis: chest X-ray image.

Output: Per task: (1) IRR: ranked list of reports or images; (2) MRG: free-text radiology report; (3) Diagnosis: multi-label binary vector over 14 disease categories.

Scoring recipe

def compute_metrics(predictions, gold):
    # R@K for retrieval
    r_at_k = sum(1 for p, g in zip(predictions, gold) if g in p[:K]) / len(predictions)
    # CIDEr for generation
    cidec = compute_tfidf_ngram_similarity(predictions, gold)
    # AUROC for classification
    auroc = mean(roc_auc_score(gold[:, c], predictions[:, c]) for c in range(14))
    return r_at_k, cidec, auroc

Common pitfalls

• MIMIC-CXR is used for both pretraining and IRR evaluation, so zero-shot and finetuning results must be reported separately to avoid data leakage confusion.
• CIDEr is explicitly marked as the main metric for report generation; relying solely on BLEU or ROUGE may misrepresent model performance.
• Diagnosis AUROC is averaged over 14 classes; class imbalance in medical datasets means macro-averaging can mask poor performance on rare conditions.

Evidence (verbatim from paper)

The R@K (recall with top k predictions) metric is used for the performance evaluation for both subtasks. ... The widely-used BLEU [26], METEOR [27], ROUGE-L [28], and CIDEr [29] for MRG are adopted as the evaluation metrics.

Citation

@misc{lin2023towardsmedical,
  title={Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining},
  author={Bingqian Lin et al.},
  year={2023},
  note={arXiv:2304.14204}
}

• arXiv: 2304.14204