Brats Robustness Eval

brats-robustness-eval

Robustness of Brain Tumor Segmentation — Müller et al. (2019) (arXiv:1912.11312, 2019)

What this evaluates

Evaluates the robustness and generalization capability of brain tumor segmentation models when faced with distribution shifts, specifically Gaussian noise perturbations in MRI scans. It probes whether high benchmark accuracy translates to reliable performance on clinically realistic, noisy data rather than just overfitting to clean benchmark distributions.

Datasets

• BraTS2018 — total 542; splits: train (285), val (66), test (191)

Metrics

• Dice score (primary) — range: [0, 1]
  • Computes the voxel-wise overlap between predicted and ground-truth segmentation masks: 2 * |intersection| / (|prediction| + |ground_truth|). Reported separately for three tumor sub-regions: enhancing tumor, complete (whole) tumor, and tumor core.

Input / output format

Input: 4-channel 3D volumetric MRI scans (T1, T1c, T2, T2-Flair), skull-stripped and spatially registered, typically at 1mm isotropic resolution.

Output: 3-channel 3D volumetric segmentation masks indicating voxel-wise labels for the enhancing tumor, complete tumor, and tumor core regions.

Scoring recipe

def compute_dice_score(predictions, ground_truth):
    # predictions and ground_truth are binary 3D masks of same shape
    intersection = np.sum(predictions * ground_truth)
    union = np.sum(predictions) + np.sum(ground_truth)
    if union == 0:
        return 1.0
    return 2.0 * intersection / union

# Applied independently per tumor sub-region (Enhancing, Complete, Core)
# Final metric is the Dice score for each region, averaged or reported separately.

Common pitfalls

• Evaluating on the official test set is restricted to a single submission, making it unsuitable for iterative robustness analysis or hyperparameter tuning.
• Assuming that higher model complexity or better benchmark Dice scores guarantee better real-world clinical generalization.
• Ignoring distribution shifts like Gaussian noise or non-isotropic voxel sizes that differ from the clean benchmark protocol.

Evidence (verbatim from paper)

In order to ensure the applicability of segmentation approaches tested on BraTS data in everyday clinical practice, it is necessary for them to show high generalization performance. For this reason, we analyze the outcomes of the different approaches when the distribution of the validation data set does not exactly match that of the training data. In a first step, we add Gaussian noise with zero mean and standard deviation $\sigma=0.02$ to the validation data. The results are depicted in Tab.2. The Dice scores indicate that the prior information about tumor appearance used in the cascadic Mumford-Shah approach is highly robust to disturbances.

Citation

@misc{muller2019robustness,
  title={Robustness of Brain Tumor Segmentation},
  author={Müller et al. (2019)},
  year={2019},
  note={arXiv:1912.11312}
}

• arXiv: 1912.11312