Medsam Laptop Eval

medsam-laptop-eval

Efficient MedSAMs: Segment Anything in Medical Images on Laptop — Ma et al. (2024) (arXiv:2412.16085, 2024)

What this evaluates

Evaluates the segmentation accuracy and inference efficiency of lightweight, promptable medical image models across diverse imaging modalities. It probes the trade-off between mask quality (overlap and boundary alignment) and computational speed on both 2D and 3D medical data.

Datasets

• MedSAMSlicer Competition Dataset — total 1933648; splits: train (1809644), test (124004); repo https://github.com/bowang-lab/MedSAMSlicer

Metrics

• Dice Similarity Coefficient (DSC) (primary) — range: [0, 1]
  • Measures the spatial overlap between the predicted segmentation mask and the ground truth mask. Calculated as 2 * |A ∩ B| / (|A| + |B|), where A and B are the sets of foreground pixels/voxels.
• Normalized Surface Distance (NSD) — range: [0, 1]
  • Evaluates boundary alignment by computing the mean distance between the surfaces of the predicted and ground truth masks, normalized by the maximum possible distance or dataset-specific scaling factor.
• Runtime — range: seconds
  • Measures the wall-clock time required to process a single image or 3D scan through the model pipeline, including encoding, prompt processing, and mask decoding.

Input / output format

Input: A medical image (2D slice or 3D volume) and a bounding box prompt specifying the target anatomical structure or lesion.

Output: A binary segmentation mask aligned with the input image dimensions, indicating the predicted target region.

Scoring recipe

def compute_dsc(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask | gt_mask)
    return (2.0 * intersection / union) if union > 0 else 0.0

def compute_nsd(pred_mask, gt_mask):
    # Compute surface distances between predicted and ground truth boundaries
    # Normalize by the maximum diameter or reference scale
    return mean_normalized_surface_distance

def compute_runtime(model, image, prompt):
    start = time.perf_counter()
    _ = model(image, prompt)
    return time.perf_counter() - start

Common pitfalls

• Models must be encapsulated in Docker containers and executed on a standardized workstation to ensure fair and consistent runtime comparisons.
• The testing set is hidden and strictly held out; participants cannot use it for tuning or validation during the competition phases.
• Runtime measurements must account for the significant computational difference between 2D slices and 3D volumetric scans, as 3D inference times show much higher variance and magnitude.

Evidence (verbatim from paper)

The teams were ranked based on both the accuracy and efficiency of their models. Specifically, the evaluation criteria included the Dice Similarity Coefficient (DSC), Normalized Surface Distance (NSD), and runtime performance.

Citation

@misc{ma2024efficientmedsam,
  title={Efficient MedSAMs: Segment Anything in Medical Images on Laptop},
  author={Ma et al. (2024)},
  year={2024},
  note={arXiv:2412.16085}
}

• arXiv: 2412.16085