Imis Eval

imis-eval

Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline — Cheng et al. (2024) (arXiv:2411.12814, 2024)

What this evaluates

Evaluates the ability of vision models to perform interactive medical image segmentation using user prompts like clicks, bounding boxes, or text. It probes how well models generalize across different imaging modalities, anatomical structures, and interaction strategies (single vs. multi-turn).

Datasets

• IMed-361M — total 6400000; splits: test (-1)
• TotalSegmentator MRI dataset — total ?; splits: test (-1)
• ISLES dataset — total ?; splits: test (-1)

Metrics

• Dice score (primary) — range: [0, 1]
  • Measures the overlap between the predicted segmentation mask and the ground truth mask. Calculated as 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground truth masks respectively.

Input / output format

Input: A medical image paired with a user prompt, which can be a single point (click), a bounding box, or a text label.

Output: A binary segmentation mask corresponding to the prompted target.

Scoring recipe

def dice_score(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = pred_mask.sum() + gt_mask.sum()
    if union == 0:
        return 1.0 if intersection == 0 else 0.0
    return 2.0 * intersection / union

Common pitfalls

• Models are evaluated on both single-interaction and multi-interaction (up to 9 turns) settings, but results are often reported only for single-turn, missing the interactive refinement capability.
• Performance heavily depends on prompt placement; points near the centroid yield significantly higher Dice scores than random placements, which can skew benchmark results if not standardized.

Evidence (verbatim from paper)

Additionally, under the single-point prompt condition, SAM and SAM-2 achieve only Dice scores of 60.26% and 59.57%, respectively, likely due to the absence of medical knowledge in the pretraining data and limited interactive information constraining model performance.

Citation

@misc{cheng2024imis,
  title={Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2411.12814}
}

• arXiv: 2411.12814