M3da Eval

m3da-eval

M3DA: Benchmark for Unsupervised Domain Adaptation in 3D Medical Image Segmentation — Boris Shirokikh et al. (arXiv:2502.17029, 2025)

What this evaluates

This benchmark evaluates unsupervised domain adaptation (UDA) methods for 3D medical image segmentation across eight practical domain shifts, including inter-modality changes (MRI-CT), scanner parameters, contrast presence, and radiation dose. It measures how well models trained on a source domain can segment target domain volumes without target labels, highlighting the robustness of adaptation techniques to real-world imaging variability.

Datasets

• AMOS — total ?; splits: (unstated)
• LIDC — total ?; splits: (unstated)
• BraTS — total ?; splits: (unstated)
• CC359 — total ?; splits: (unstated)

Metrics

• multiclass Dice score (primary) — range: [0, 1]
  • Intersection over Union between predicted and ground truth segmentation masks, averaged across all foreground classes. The background label is explicitly excluded from quantification.
• percentage of performance gap closed — range: percent
  • 100 * (Method_Dice - Baseline_Dice) / (Oracle_Dice - Baseline_Dice). Measures how much of the performance gap between a zero-shot baseline and an Oracle model (trained on target domain) is closed by the adaptation method.

Input / output format

Input: 3D medical image volumes (CT, MRI, LDCT, etc.) from a source domain with corresponding segmentation masks for training; target domain volumes for evaluation.

Output: Multiclass segmentation masks (excluding background) for each target domain volume.

Scoring recipe

def dice_score(pred, gt):
    intersection = np.sum(pred[gt == 1])
    union = np.sum(pred) + np.sum(gt)
    return 2 * intersection / union if union > 0 else 0.0

def compute_metrics(preds, gold, baseline_preds, oracle_preds):
    dice = np.mean([dice_score(p, g) for p, g in zip(preds, gold)])
    b_dice = np.mean([dice_score(p, g) for p, g in zip(baseline_preds, gold)])
    o_dice = np.mean([dice_score(p, g) for p, g in zip(oracle_preds, gold)])
    gap = 100 * (dice - b_dice) / (o_dice - b_dice) if (o_dice - b_dice) != 0 else 0.0
    return dice, gap

Common pitfalls

• The background label is explicitly excluded from the Dice score calculation; including it will artificially lower scores.
• The 'Oracle' setup refers to training and validating directly on the target domain, not a separate pre-trained model.
• Gap closed scores can be negative if an adaptation method performs worse than the zero-shot baseline.
• Results report case-wise standard deviations in parentheses, indicating multiple runs or cross-validation folds rather than a single deterministic evaluation.

Evidence (verbatim from paper)

We evaluated various DA methods on the M3DA benchmark (Table 5) using multi-class Dice score and the percentage of performance gap closed between the Baseline and Oracle setups: $100 \times \frac{\text{Method}{\text{Dice}} - \text{Baseline}{\text{Dice}}}{\text{Oracle}{\text{Dice}} - \text{Baseline}{\text{Dice}}}$ .

Citation

@misc{shirokikh2025m3da,
  title={M3DA: Benchmark for Unsupervised Domain Adaptation in 3D Medical Image Segmentation},
  author={Boris Shirokikh et al.},
  year={2025},
  note={arXiv:2502.17029}
}

• arXiv: 2502.17029