Occsam Bench Eval

occsam-bench-eval

Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models — Ho et al. (2026) (arXiv:2604.11711, 2026)

What this evaluates

Evaluates the robustness of foundation segmentation models to synthetic surgical tool occlusions in endoscopic images. It probes whether models accurately segment visible tissue, avoid hallucinating into occluded regions, or maintain amodal completion under varying occlusion severities and prompt types.

Datasets

• CVC-300 — total ?; splits: test (-1)
• CVC-ColonDB — total ?; splits: test (-1)
• ETIS — total ?; splits: test (-1)

Metrics

• DSC (primary) — range: [0, 1]
  • Dice Similarity Coefficient: $2|A \cap B| / (|A| + |B|)$, where A and B are the predicted and ground truth binary masks respectively.
• HD95 — range: pixels or mm
  • 95th percentile of the Hausdorff distance between the predicted and ground truth boundaries.
• Relative Degradation — range: percent
  • Relative performance drop from clean to occluded conditions: $(\text{DSC}{\text{clean}}-\text{DSC}{\text{occ}})/\text{DSC}_{\text{clean}}\times 100$.

Input / output format

Input: Endoscopic image with synthetic surgical tool occlusion (or cutout) at varying severity levels (Clean, Low, Medium, High), paired with a box or single-point prompt indicating the target polyp.

Output: Binary segmentation mask of the target polyp.

Scoring recipe

def compute_dsc(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask | gt_mask)
    return 2 * intersection / union if union > 0 else 0.0

def compute_relative_degradation(dsc_clean, dsc_occ):
    return (dsc_clean - dsc_occ) / dsc_clean * 100

# Evaluate across occlusion types, severities, and prompt types
for model in models:
    for dataset in datasets:
        for occlusion_type in ['cutout', 'surgical_tool']:
            for severity in ['clean', 'low', 'med', 'high']:
                for prompt in ['box', 'point']:
                    pred = model.predict(image, prompt)
                    dsc = compute_dsc(pred, gt)
                    if severity == 'clean':
                        dsc_clean = dsc
                    else:
                        delta = compute_relative_degradation(dsc_clean, dsc)
                        report(dsc, delta)

Common pitfalls

• High full-mask (amodal) DSC can be misleading if the model predicts into the occluded tool region rather than around it, masking clinically dangerous hallucinations.
• High invisible-region DSC at severe occlusion levels is confounded by geometric coincidence due to large tool footprints, not necessarily indicating true amodal completion.
• MedSAM is trained exclusively on box prompts; evaluating it with point prompts causes systematic failure and should be excluded from comparative analysis.

Evidence (verbatim from paper)

Relative degradation is computed as $\Delta(%)=(\text{DSC}{\text{clean}}-\text{DSC}{\text{occ}})/\text{DSC}_{\text{clean}}\times 100$. Unless stated otherwise, we report box prompt results under surgical tool occlusion as the primary condition, as it is the most clinically motivated and reveals the sharpest behavioral differences. All results are reported in DSC; HD95 results are included in supplementary material.

Citation

@misc{ho2026occsam,
  title={Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models},
  author={Ho et al. (2026)},
  year={2026},
  note={arXiv:2604.11711}
}

• arXiv: 2604.11711