Kits21 Segmentation Eval

kits21-segmentation-eval

The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT — Heller et al. (2023) (arXiv:2307.01984, 2023)

What this evaluates

This benchmark evaluates the ability of deep learning models to perform multi-organ and multi-lesion semantic segmentation on 3D medical imaging data. It specifically probes a model's capacity to accurately delineate kidneys, renal tumors, and renal cysts from corticomedullary-phase CT scans, testing both volumetric overlap and boundary precision.

Datasets

• KiTS21 — total ?; splits: test (100)

Metrics

• dice (primary) — range: [0, 1]
  • Volumetric Dice similarity coefficient: 2 * |prediction ∩ ground_truth| / (|prediction| + |ground_truth|). The final ranking uses the mean across all test cases.
• surface_dice — range: [0, 1]
  • Surface Dice: Fraction of surface points in the prediction within a specified distance threshold of the ground truth surface. Also averaged across all test cases for ranking.

Input / output format

Input: 3D corticomedullary-phase CT scan volumes (voxel data).

Output: 3D segmentation masks containing binary labels for kidneys, renal tumors, and renal cysts.

Scoring recipe

def score_kits21(predictions, ground_truths):
    dice_scores = [dice_score(p, g) for p, g in zip(predictions, ground_truths)]
    surface_dice_scores = [surface_dice_score(p, g) for p, g in zip(predictions, ground_truths)]
    avg_dice = np.mean(dice_scores)
    avg_surface_dice = np.mean(surface_dice_scores)
    return {'dice': avg_dice, 'surface_dice': avg_surface_dice}
# Final ranking uses a rank-then-aggregate procedure on the mean scores across all test cases.

Common pitfalls

• The leaderboard ranking is determined by a rank-then-aggregate procedure using both volumetric and surface Dice scores, not a single metric.
• A higher leaderboard rank does not guarantee statistical superiority; the authors explicitly recommend bootstrap sampling and Holm-Bonferroni corrections to assess performance differences.
• Ground truth is derived from three independent expert annotations per region, so models must align with a multi-annotator consensus protocol rather than a single label.

Evidence (verbatim from paper)

The final leaderboard ranking was determined with a rank-then-aggregate procedure using the respective means across HECs of the two chosen varieties of dice scores.

Citation

@misc{heller2023kits21,
  title={The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT},
  author={Heller et al. (2023)},
  year={2023},
  note={arXiv:2307.01984}
}

• arXiv: 2307.01984