Qu Brats Score

qu-brats-score

QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results — Mehta et al. (2021) (arXiv:2112.10074, 2021)

What this evaluates

Evaluates the calibration of voxel-wise uncertainty estimates in brain tumor segmentation. It measures how effectively uncertainty thresholds filter out incorrect predictions while preserving correct ones, rewarding high confidence in accurate regions and penalizing the loss of correct predictions when filtering uncertain voxels.

Datasets

• BraTS — total ?; splits: train (228), val (57), test (50); repo https://github.com/RagMeh11/QU-BraTS

Metrics

• QU-BraTS unified score (primary) — range: [0, 1]
  • Computes the Area Under the Curve (AUC) for three metrics across uncertainty thresholds τ: Dice Similarity Coefficient (DSC), Filtered True Positives ratio (FTP), and Filtered True Negatives ratio (FTN). FTP = (TP_100 - TP_τ) / TP_100, FTN = (TN_100 - TN_τ) / TN_100. The final score is (AUC_DSC + (1 - AUC_FTP) + (1 - AUC_FTN)) / 3.
• DSC — range: [0, 1]
  • Dice Similarity Coefficient measuring voxel-wise overlap between predicted and ground truth segmentation masks.

Input / output format

Input: 3D MRI volumes with ground truth binary masks for tumor entities (ET, TC, WT) and predicted voxel-wise uncertainty maps normalized to [0, 100].

Output: Predicted segmentation masks and corresponding voxel-wise uncertainty values for each tumor entity.

Scoring recipe

def compute_qu_brats_score(pred_masks, gt_masks, uncertainty_maps, thresholds=[25, 50, 75, 100]):
    dsc_vals, ftp_vals, ftm_vals = [], [], []
    for tau in thresholds:
        filtered_pred = mask_where_uncertainty_lt(pred_masks, uncertainty_maps, tau)
        dsc_vals.append(dice_score(filtered_pred, gt_masks))
        tp_100 = count_true_positives(pred_masks, gt_masks)
        tp_tau = count_true_positives(filtered_pred, gt_masks)
        tn_100 = count_true_negatives(pred_masks, gt_masks)
        tn_tau = count_true_negatives(filtered_pred, gt_masks)
        ftp_vals.append((tp_100 - tp_tau) / tp_100)
        ftm_vals.append((tn_100 - tn_tau) / tn_100)
    auc_dsc = np.trapz(dsc_vals, thresholds)
    auc_ftp = np.trapz(ftp_vals, thresholds)
    auc_ftn = np.trapz(ftm_vals, thresholds)
    return (auc_dsc + (1 - auc_ftp) + (1 - auc_ftn)) / 3

Common pitfalls

• Filtering out correct predictions (TPs/TNs) to artificially inflate DSC is heavily penalized by the FTP/FTN ratios.
• Uncertainty maps must be strictly normalized to the 0-100 range before applying thresholds.
• Evaluation is performed separately for each tumor entity (ET, TC, WT) due to clinical relevance and class imbalance.

Evidence (verbatim from paper)

Finally, the resulting uncertainty measures for each team are ranked according to a unified score which combines the area under three curves: 1) DSC vs τ, 2) FTP vs τ, and 3) FTN vs τ, for different values of τ. The unified score is calculated as follows: score_tumor_entity = (AUC1 + (1 - AUC2) + (1 - AUC3)) / 3.

Citation

@misc{mehta2021qu_brats,
  title={QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results},
  author={Mehta et al. (2021)},
  year={2021},
  note={arXiv:2112.10074}
}

• arXiv: 2112.10074