Ood Detection Histopathology Eval

ood-detection-histopathology-eval

Uncertainty estimation for out-of-distribution detection in computational histopathology — Goetz (2022) (arXiv:2210.09909, 2022)

What this evaluates

Evaluates the ability of uncertainty estimation methods to distinguish in-distribution clinical histopathology samples from out-of-distribution samples and to provide well-calibrated confidence scores for selective prediction. It probes how different methods maintain predictive accuracy and calibration under varying degrees of data shift.

Datasets

• CIFAR-10 — total ?; splits: train (-1), test (-1)
• Hospital 4 — total ?; splits: test (-1)
• Hospital 5 — total ?; splits: test (-1)
• Cancer type — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified samples among those retained after uncertainty thresholding.
• Youden's J — range: [-1, 1]
  • Threshold optimization criterion for selective prediction, calculated as sensitivity + specificity - 1.
• average precision — range: [0, 1]
  • Area under the precision-recall curve, summarizing model performance across all classification thresholds.

Input / output format

Input: Image patches (histopathology slides) or standard images (CIFAR-10) passed through a neural network feature extractor.

Output: Predicted class label, class probability distribution, and a scalar uncertainty score (e.g., predictive variance or distance-based uncertainty).

Scoring recipe

def compute_metrics(predictions, labels, uncertainty_scores, threshold):
    accepted_mask = uncertainty_scores < threshold
    acc = mean(predictions[accepted_mask] == labels[accepted_mask])
    retention_rate = mean(accepted_mask)
    return acc, retention_rate

# Threshold optimization via Youden's J
# J = sensitivity + specificity - 1
# threshold = argmax_J(J) on validation set

Common pitfalls

• Setting uncertainty thresholds on the same data used for evaluation, which causes data leakage and inflates OOD detection performance.
• Assuming far OOD detection is inherently easier than near OOD detection; performance degrades sharply as data shift increases.
• Ignoring model calibration; high accuracy does not guarantee reliable uncertainty estimates for clinical deployment.

Evidence (verbatim from paper)

Table 4: Fraction of samples (mean ± std) retained after rejection based on an uncertainty threshold that optimized Youden's J, set on hospital 5 and hospital 4 for hospital 4 and hospital 5 & cancer type, respectively. ... While uncertainty thresholding does in general increase accuracy, the gains are very variable across datasets and methods.

Citation

@misc{goetz2022uncertainty,
  title={Uncertainty estimation for out-of-distribution detection in computational histopathology},
  author={Goetz (2022)},
  year={2022},
  note={arXiv:2210.09909}
}

• arXiv: 2210.09909