Medsam2 Segmentation Eval

medsam2-segmentation-eval

MedSAM2: Segment Anything in 3D Medical Images and Videos — Ma et al. (2025) (arXiv:2504.03600, 2025)

What this evaluates

Evaluates promptable 3D medical image and video segmentation across diverse organs, lesions, and imaging modalities. It probes spatial consistency across 3D slices and temporal continuity across video frames using bounding box prompts.

Datasets

• Holdout 3D Test Set — total 2309; splits: test (2309)
• CAMUS — total 100; splits: test (100)
• SUN — total 173; splits: test (173)

Metrics

• Dice similarity coefficient (DSC) (primary) — range: [0, 1]
  • DSC = 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground truth binary masks. Results are reported as the median DSC across tasks/instances with interquartile ranges (IQR).

Input / output format

Input: 2D medical image slices or video frames paired with prompt embeddings. For 3D images, a bounding box prompt is provided on the middle slice. For videos, prompts are provided on the first frame.

Output: A 2D segmentation mask for the prompted slice/frame. For 3D volumes, the model propagates the 2D mask bidirectionally to generate a full 3D segmentation mask.

Scoring recipe

def compute_dsc(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    if union == 0:
        return 1.0
    return 2.0 * intersection / union

dsc_scores = [compute_dsc(pred, gt) for pred, gt in zip(predictions, ground_truths)]
median_dsc = np.median(dsc_scores)
iqr_dsc = np.percentile(dsc_scores, 75) - np.percentile(dsc_scores, 25)

Common pitfalls

• The evaluation uses a specific bidirectional propagation protocol: the model generates a 2D mask at the middle slice and propagates it to the top and bottom slices, rather than processing the entire volume slice-by-slice independently.
• Results are aggregated as median DSC with IQR across 40 distinct tasks, not as a simple mean accuracy across all individual cases.
• Bounding boxes on the middle slice are the standard prompt; using points or masks will yield different results and break protocol consistency.

Evidence (verbatim from paper)

We first evaluated the trained model on the holdout 3D test set, which contains 40 segmentation tasks from different cohorts across a wide range of organs and lesions in CT, MRI, and PET scans. ... Fig.[2]a shows the quantitative results on the 3D testing set ... MedSAM2 consistently achieves the highest DSC scores across all targets ... Next, we evaluated video segmentation performance for heart chambers and polyp segmentation in cardiac ultrasound (Echocardiography) and endoscopy videos on the widely used CAMUS and SUN datasets ... MedSAM2 achieves better performance across the three tasks, with the highest DSC scores of 96.13% ... 93.10% ... 95.79% ... On the hard polyp subset ... MedSAM2 outperforms SAM2.1 with a noticeable gap and more consistent DSC scores of 92.22%.

Citation

@misc{ma2025medsam2,
  title={MedSAM2: Segment Anything in 3D Medical Images and Videos},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2504.03600}
}

• arXiv: 2504.03600