Sam Zero Shot Medical Eval

sam-zero-shot-medical-eval

Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation — Peilun Shi et al. (2023) (arXiv:2304.12637, 2023)

What this evaluates

Evaluates the zero-shot segmentation capability of the Segment Anything Model (SAM) across diverse medical imaging modalities and anatomical structures. It probes the model's ability to generalize without task-specific fine-tuning to structured medical targets like organs, lesions, and retinal layers.

Datasets

• Skin Lesion Analysis Toward Melanoma Detection (ISIC) — total 1000; splits: test (1000)
• DoFE (Drishiti-GS, RIM-ONE-r3, REFUGE amalgamated) — total 271; splits: test (271)
• AMOS — total 15361; splits: val (15361)
• MICCAI 2017 Robotic Instrument Segmentation — total 1200; splits: test (1200)
• Chest X-ray — total 704; splits: test (704)
• Rat Colon — total 130; splits: test (130)
• AROI — total 113; splits: test (113)

Metrics

• Dice similarity coefficient (primary) — range: [0, 1]
  • Measures overlap between ground truth Y and predicted mask Ŷ: Dice = 2|Y ∩ Ŷ| / (|Y| + |Ŷ|). Ranges from 0 (no overlap) to 1 (perfect overlap). Averaged across samples.
• Intersection over Union (IoU) — range: [0, 1]
  • Computes ratio of intersection to union of two sets: IoU = |Y ∩ Ŷ| / (|Y| + |Ŷ| - |Y ∩ Ŷ|). Ranges from 0 to 1. Averaged across samples.

Input / output format

Input: Single medical image (dermoscopy, fundus, CT, MRI, endoscopy, X-ray, or OCT) with a point prompt corresponding to the centroid of the ground truth mask.

Output: Binary segmentation mask (predicted mask Ŷ) of the target anatomical structure or lesion.

Scoring recipe

for each image in dataset:
  Y = ground_truth_mask
  Y_hat = sam_predict(image, prompt=centroid(Y))
  dice = 2 * np.sum(Y * Y_hat) / (np.sum(Y) + np.sum(Y_hat))
  iou = np.sum(Y * Y_hat) / (np.sum(Y) + np.sum(Y_hat) - np.sum(Y * Y_hat))
  store dice, iou
avg_dice = np.mean(all_dices)
avg_iou = np.mean(all_ious)

Common pitfalls

• SAM is prompted with the ground-truth centroid, which assumes oracle knowledge and does not reflect realistic user interaction or automated prompt generation.
• Some datasets (e.g., AMOS) lack released test labels, forcing the use of validation sets for evaluation, which may inflate or bias results.
• Small sample sizes in certain subsets (e.g., 259 vs 1000 for ISIC) lead to high variance in reported scores.
• SAM's zero-shot performance heavily depends on image contrast rather than semantic understanding, particularly in fundus images, making cross-domain comparisons sensitive to lighting/contrast variations.

Evidence (verbatim from paper)

To assess the zero-shot segmentation capability of SAM on medical images, two quantitative metrics were employed: Dice similarity coefficient and Intersection over Union (IoU). The Dice coefficient measures the overlap between two sets of data and ranges from 0 (no overlap) to 1 (perfect overlap). Similarly, IoU computes the ratio of the intersection over the union of two sets and ranges from 0 to 1. Both metrics were then averaged across multiple samples to obtain an overall measure of the segmentation accuracy.

Citation

@misc{shi2023generalist,
  title={Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation},
  author={Peilun Shi et al. (2023)},
  year={2023},
  note={arXiv:2304.12637}
}

• arXiv: 2304.12637