Segbook Eval

segbook-eval

SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation — Jin Ye et al. (2024) (arXiv:2411.14525, 2024)

What this evaluates

Evaluates the transfer learning and fine-tuning capabilities of volumetric medical image segmentation models across diverse imaging modalities, anatomical targets, and dataset sizes. It probes how well pre-trained models generalize to downstream segmentation tasks in clinical imaging scenarios and reveals non-linear performance scaling with dataset scale.

Datasets

• SegBook — total 87; splits: (unstated)

Metrics

• Dice Score (DSC) (primary) — range: percent
  • Computes the overlap between predicted and ground-truth segmentation masks: DSC = 2|X ∩ Y| / (|X| + |Y|). Reported as a percentage (0–100) averaged across datasets or stratified by modality, target type, and dataset size.

Input / output format

Input: 3D volumetric medical images (CT, MRI, PET, or Ultrasound) with corresponding segmentation masks for fine-tuning; evaluation uses the test/validation splits of 87 downstream datasets.

Output: 3D volumetric segmentation masks predicting the target anatomical structures or lesions.

Scoring recipe

def compute_dice(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask) + np.sum(gt_mask)
    if union == 0:
        return 100.0 if intersection == 0 else 0.0
    return 200.0 * intersection / union

# Average across all 87 datasets or specific subsets (e.g., by modality or size)
final_score = np.mean([compute_dice(pred, gt) for pred, gt in zip(predictions, golds)])

Common pitfalls

• The paper reports DSC on a 0–100 scale, not 0–1, which can cause confusion when comparing to other benchmarks that use fractional values.
• Performance exhibits a non-linear bottleneck effect: fine-tuning yields ~3% gains on small and large datasets but only ~1% on medium-sized datasets, so averaging across scales without stratification masks this behavior.
• Evaluations are heavily stratified by modality (CT, MRI, US, CT&PET) and target type (seen/unseen structures, lesions, bone, vessel); results must be reported per stratum to avoid misleading cross-task averages.

Evidence (verbatim from paper)

To investigate the overall transfer learning performance of STU-Net, we conducted experiments on 87 downstream datasets. As shown in Table 2, we reported average DSC for various models and detailed DSC for three data scales: small (S), medium (M), and large (L).

Citation

@misc{ye2024segbook,
  title={SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation},
  author={Jin Ye et al. (2024)},
  year={2024},
  note={arXiv:2411.14525}
}

• arXiv: 2411.14525