Uncertainty Estimation Benchmark Eval

uncertainty-estimation-benchmark-eval

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise — Mehrtens et al. (2023) (arXiv:2301.01054, 2023)

What this evaluates

Evaluates the robustness, calibration, and selective classification capability of uncertainty estimation methods (Deep Ensembles, MC Dropout, SVI, TTA) on histopathological whole slide images under domain shift and label noise.

Datasets

• Camelyon17 — total ?; splits: train (-1), test (-1); repo https://github.com/DBO-DKFZ/uncertainty-benchmark
• TCGA — total ?; splits: train (-1), test (-1)

Metrics

• AUARC (primary) — range: [0, 1]
  • Area Under the Accuracy-Reject Curve. Computed by iteratively rejecting the most uncertain tiles (sorted by 1-confidence) and calculating the balanced accuracy on the remaining set, then integrating the resulting curve.
• ECE — range: [0, 1]
  • Expected Calibration Error. Computed as the median calibration error over all slides, measuring the discrepancy between predicted confidence and actual accuracy across confidence bins.
• Balanced Accuracy — range: [0, 1]
  • Average of recall obtained on each class. Used to evaluate classification performance while accounting for class imbalance.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Used for slide-level MSI status prediction to evaluate ranking performance between positive and negative classes.

Input / output format

Input: Histopathological whole slide image (WSI) tiles or whole slides with associated tumor annotations and binary labels (tumor/non-tumor or MSS/MSI).

Output: Per-tile or per-slide prediction confidence/probability and binary class label. Slide-level predictions are derived by aggregating tile-level scores (e.g., averaging top 1% confident tiles or using CLAM attention).

Scoring recipe

def compute_auarc(predictions, labels, uncertainty_scores):
    sorted_indices = np.argsort(uncertainty_scores)[::-1]
    accuracies = []
    for k in range(len(predictions)):
        remaining = sorted_indices[k:]
        acc = balanced_accuracy(labels[remaining], predictions[remaining])
        accuracies.append(acc)
    return np.trapz(accuracies, np.linspace(0, 1, len(accuracies)))

def compute_ece(predictions, labels, confidences, n_bins=15):
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for i in range(n_bins):
        mask = (confidences > bin_boundaries[i]) & (confidences <= bin_boundaries[i+1])
        if np.sum(mask) > 0:
            bin_acc = np.mean(labels[mask] == predictions[mask])
            bin_conf = np.mean(confidences[mask])
            ece += np.sum(mask) * abs(bin_acc - bin_conf)
    return ece / len(labels)

Common pitfalls

• Assuming ensembles always improve calibration; they boost accuracy and rejection performance but may not reduce ECE under label noise.
• Confusing tile-level uncertainty aggregation with slide-level methods like CLAM; simple averaging of top confident tiles underperforms attention-based aggregation.
• Expecting OOD centers to always have lower accuracy than ID centers; the paper notes OOD balanced accuracy can be partially higher due to dataset characteristics.

Evidence (verbatim from paper)

In [Table 1] we show the area under the curve for the accuracy-reject curves (AUARC) for the weak and strong domain shift scenarios. ... In the plot on the right-hand side of Figure 1, we evaluate model calibration in terms of ECE (see Section 2.3) for the weak and strong shift. The ECE values have been computed as the median calibration error over all slides...

Citation

@misc{mehrtens2023uncertaintybenchmark,
  title={Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise},
  author={Mehrtens et al. (2023)},
  year={2023},
  note={arXiv:2301.01054}
}

• arXiv: 2301.01054