Cold Start Al 3d Medical Seg Eval

cold-start-al-3d-medical-seg-eval

COLosSAL: A Benchmark for Cold-start Active Learning for 3D Medical Image Segmentation — Liu et al. (2023) (arXiv:2307.12004, 2023)

What this evaluates

Evaluates cold-start active learning sample selection strategies for 3D medical image segmentation by comparing how well diversity-based, uncertainty-based, and random methods perform when annotation budgets are extremely limited.

Datasets

• Medical Segmentation Decathlon (MSD) — total ?; splits: test (-1); repo https://github.com/MedICL-VU/COLosSAL

Metrics

• Dice score (primary) — range: [0, 1]
  • 2 * |A ∩ B| / (|A| + |B|), measuring the voxel-wise overlap between the predicted segmentation mask and the ground truth mask.
• HD95 — range: mm
  • 95th percentile of the Hausdorff distance between the predicted and ground truth boundary surfaces, measuring the worst-case boundary localization error.

Input / output format

Input: 3D medical image volumes (and optionally local ROIs) used as input for cold-start active learning sample selection strategies.

Output: Segmentation masks for each selected volume, evaluated against ground truth to compute Dice and HD95 scores.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    dice = 2.0 * intersection / (np.sum(pred_mask) + np.sum(gt_mask))
    # HD95 computed via distance transforms
    dist_map = hausdorff_distance_transform(pred_mask, gt_mask)
    hd95 = np.percentile(dist_map, 95)
    return dice, hd95

Common pitfalls

• Assuming active learning strategies consistently outperform random selection; the benchmark shows random selection remains a strong baseline in cold-start scenarios.
• Expecting local ROI-based selection to outperform global volume-based selection; results show local strategies fail to consistently surpass global ones.
• Overestimating performance on tumor-containing tasks; uncertainty-based methods struggle significantly here due to poor uncertainty estimation on pathological regions.

Evidence (verbatim from paper)

In Fig. 2, with a fixed budget of 5 samples (except for Heart, where 3 samples are used), we compare the uncertainty-based and diversity-based strategies against the random selection on five different segmentation tasks. Note that the selections made by each of our evaluated AL strategies are deterministic. For random selection, we visualize the individual Dice scores (red dots) of all 15 runs as well as their mean (dashed line). HD95 results (Supp. Tab. 1) follow the same trends.

Citation

@misc{liu2023colossal,
  title={COLosSAL: A Benchmark for Cold-start Active Learning for 3D Medical Image Segmentation},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2307.12004}
}

• arXiv: 2307.12004