Medsg Bench Eval

medsg-bench-eval

MedSG-Bench: A Benchmark for Medical Image Sequences Grounding — Jingkun Yue et al. (2025) (arXiv:2505.11852, 2025)

What this evaluates

This benchmark evaluates sequential visual grounding in medical imaging, specifically testing a model's ability to perform cross-image semantic alignment, detect differences between sequential scans, and identify consistent regions across time.

Datasets

• MedSG-Bench — total 9630; splits: test (9630)

Metrics

• average IoU (primary) — range: [0, 1]
  • Intersection over Union between the predicted grounding region (bounding box or mask) and the ground truth region, averaged across all samples.
• ACC@0.5 — range: [0, 1]
  • Accuracy thresholded at IoU=0.5, representing the fraction of samples where the predicted region achieves an IoU of at least 0.5 with the ground truth.

Input / output format

Input: A sequence of medical images (multiple modalities/views) paired with a text prompt/question requiring spatial localization or difference/consistency detection.

Output: Predicted bounding box coordinates or segmentation mask indicating the grounded region in the image sequence.

Scoring recipe

def compute_metrics(predictions, golds):
    ious = []
    for pred, gold in zip(predictions, golds):
        intersection = len(pred & gold)
        union = len(pred | gold)
        ious.append(intersection / union if union > 0 else 0.0)
    avg_iou = sum(ious) / len(ious)
    acc_05 = sum(1 for i in ious if i >= 0.5) / len(ious)
    return avg_iou, acc_05

Common pitfalls

• Evaluations must be strictly zero-shot; providing in-context examples or fine-tuning on the benchmark violates the protocol.
• Medical-domain specialized models often underperform general-purpose models on this task due to catastrophic forgetting of spatial grounding capabilities during domain-specific instruction tuning.
• Model scale and release recency do not guarantee better grounding performance; many recent models are optimized for high-level semantic tasks rather than fine-grained spatial alignment.

Evidence (verbatim from paper)

In this study, we evaluate model performance under a zero-shot setting, where the models were prompted to perform inference without access to in-context examples. We use average Intersection over Union (IoU) and ACC@0.5 as the evaluation metric.

Citation

@misc{yue2025medsgbench,
  title={MedSG-Bench: A Benchmark for Medical Image Sequences Grounding},
  author={Jingkun Yue et al. (2025)},
  year={2025},
  note={arXiv:2505.11852}
}

• arXiv: 2505.11852