Crossmed Eval

crossmed-eval

CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging — Pooja Singh et al. (2025) (arXiv:2511.11034, 2025)

What this evaluates

Evaluates compositional generalization in medical vision-language models across a structured Modality–Anatomy–Task (MAT) schema. It probes zero-shot cross-task transfer, generalization to novel MAT combinations, and robustness under low-data regimes using a unified visual question answering interface.

Datasets

• CrossMed — total 20200; splits: train (-1), val (-1), test (-1)

Metrics

• top-1 classification accuracy (primary) — range: percent
  • Fraction of correctly predicted class labels out of total test instances, multiplied by 100.
• segmentation class-wise Intersection-over-Union (cIoU) — range: [0, 1]
  • Average Intersection-over-Union computed across all segmentation classes: IoU = |prediction ∩ ground_truth| / |prediction ∪ ground_truth|.

Input / output format

Input: Medical imaging modality (chest X-ray, brain MRI, or chest CT) paired with a structured visual question answering (VQA) prompt following the Modality–Anatomy–Task schema.

Output: For classification: a discrete multi-choice class label. For segmentation: a selected mask option from a VQA-style multi-choice head.

Scoring recipe

def score(predictions, gold):
    # Classification accuracy
    correct = sum(1 for p, g in zip(predictions['class'], gold['class']) if p == g)
    acc = (correct / len(gold['class'])) * 100
    
    # Segmentation cIoU
    ious = []
    for pred_mask, gold_mask in zip(predictions['seg'], gold['seg']):
        inter = np.logical_and(pred_mask, gold_mask).sum()
        union = np.logical_or(pred_mask, gold_mask).sum()
        ious.append(inter / union if union > 0 else 0.0)
    cIoU = np.mean(ious)
    return acc, cIoU

Common pitfalls

• Misinterpreting MAT evaluation splits: 'Related' shares two triplet elements, 'Unrelated' shares at most one, and 'Zero-Overlap' shares none; performance drops drastically between them.
• Confusing evaluation heads: The benchmark's primary protocol uses a VQA-style multi-choice head for segmentation to enable cross-task comparison, not a standard continuous U-Net decoder.
• Assuming standard random splits: The benchmark requires leave-one-triplet-out and zero-overlap splits that strictly factorize Modality, Anatomy, and Task, rather than random image-level splits.

Evidence (verbatim from paper)

We present a comprehensive evaluation of the CrossMed benchmark under five key axes: in-domain multi-task learning, compositional generalization, low-data regimes, cross-task transfer, and architectural comparisons. Our work follows a unified VQA formulation, testing generalization across Related/Unrelated MAT splits and measuring top-1 classification accuracy and segmentation class-wise Intersection-over-Union( cIoU).

Citation

@misc{singh2025crossmed,
  title={CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging},
  author={Pooja Singh et al. (2025)},
  year={2025},
  note={arXiv:2511.11034}
}

• arXiv: 2511.11034