Brats2017 Seg Eval

brats2017-seg-eval

Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation — Kamnitsas et al. (2017) (arXiv:1711.01468, 2017)

What this evaluates

Evaluates the capability of deep learning models to perform multi-modal brain tumour segmentation from 3D MRI volumes. It specifically probes robustness to architectural choices, loss functions, and intensity normalization preprocessing pipelines by aggregating diverse model configurations.

Datasets

• BRATS 2017 — total 477; splits: train (285), val (46), test (146)

Metrics

• Dice score (DSC) (primary) — range: percent
  • Computes the overlap between predicted and ground truth segmentation masks: 2|A∩B| / (|A|+|B|). Reported as a percentage across merged tumour regions (whole, core, enhancing).
• Sensitivity — range: percent
  • True positive rate: TP / (TP + FN). Measures the proportion of actual tumour voxels correctly identified by the model.
• Hausdorff_95 — range: other
  • 95th percentile of the Hausdorff distance between the boundaries of the predicted and ground truth masks, providing a robust measure of maximum boundary deviation that is less sensitive to outliers than the standard Hausdorff distance.

Input / output format

Input: 4-channel 3D MRI volumes (FLAIR, T1, T1ce, T2), pre-processed by organisers to be skull-stripped, registered to a common space, and resampled to isotropic 1mm³ resolution (240×240×155 voxels).

Output: 3-channel 3D label maps per voxel corresponding to: 1) necrotic core and non-enhancing tumour, 2) oedema, 4) enhancing core. (Label 3 is unused). Outputs are evaluated as merged anatomical sets: whole tumour, core, and enhancing tumour.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask | gt_mask)
    dsc = 2 * intersection / union if union > 0 else 0.0
    tp = np.sum(pred_mask & gt_mask)
    fn = np.sum((~pred_mask) & gt_mask)
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    # Hausdorff_95: 95th percentile of pairwise boundary distances
    boundaries_pred = extract_boundary(pred_mask)
    boundaries_gt = extract_boundary(gt_mask)
    dists = pairwise_distances(boundaries_pred, boundaries_gt)
    hd95 = np.percentile(dists, 95)
    return dsc, sensitivity, hd95

Common pitfalls

• Validation and test set ground truths are hidden; evaluation must be submitted via an online challenge system with a strict 48-hour window for the test phase.
• Intensity normalization significantly impacts performance; models trained on different preprocessing pipelines must be ensembled to achieve robust results.
• Only three tumour labels are used (1, 2, 4); label 3 is explicitly ignored, and metrics are computed on merged anatomical sets (whole, core, enhancing).
• Single submission allowed per team during the test phase, making hyperparameter tuning and model selection critical before submission.

Evidence (verbatim from paper)

Our system won the competition by achieving the overall best performance in the testing phase, based on Dice score (DSC) and Haussdorf distance. We also show results achieved on the validation set by the teams that ranked in the next two positions at the testing stage.

Citation

@misc{kamnitsas2017ensembles,
  title={Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation},
  author={Kamnitsas et al. (2017)},
  year={2017},
  note={arXiv:1711.01468}
}

• arXiv: 1711.01468