Camelyon17 Wilds Eval

camelyon17-wilds-eval

PathoWAve: A Deep Learning-based Weight Averaging Method for Improving Domain Generalization in Histopathology Images — Parastoo Sotoudeh Sharifi, M. Omair Ahmad, M.N.S. Swamy (2024) (arXiv:2406.15685, 2024)

What this evaluates

Evaluates domain generalization capability for metastatic breast cancer detection in histopathology images. The protocol trains models on data from known medical centers and tests them on completely unseen centers with different staining protocols and scanners to measure adaptability.

Datasets

• Camelyon17 WILDS — total ?; splits: train (302436), id_val (33560), val (34904), test (85054)

Metrics

• accuracy (%) (primary) — range: percent
  • Classification accuracy calculated as the percentage of correctly predicted patches out of the total number of patches in the validation or test set.

Input / output format

Input: Image patches from Whole Slide Images of lymph node sections, labeled for metastatic breast cancer presence.

Output: Binary classification prediction (metastatic vs. non-metastatic) or probability scores, evaluated as a percentage accuracy.

Scoring recipe

def compute_accuracy(predictions, labels):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    return (correct / len(labels)) * 100

Common pitfalls

• Models are evaluated on completely unseen medical centers, not random splits, so results reflect domain generalization rather than standard IID performance.
• The validation set is used for hyperparameter tuning and model selection, while the test set is held out until the final evaluation, mirroring the WILDS protocol.

Evidence (verbatim from paper)

This dataset is partitioned by medical center origin, for developing generalized models to unseen data for cancerous tissues detection. For training and identification validation (id val), data come from three hospitals (30 WSIs and 302,436 patches for training, plus 33,560 patches for id val), while validation (val) and testing datasets are sourced from unique, previously unseen hospitals—val with 10 WSIs and 34,904 patches from one hospital, and testing with 10 WSIs and 85,054 patches from another hospital... TABLE I COMPARATIVE PERFORMANCE OF VARIOUS DOMAIN GENERALIZATION METHODS ON VALIDATION AND TEST SET OF CAMELYON17 WILDS DATASET. Validation % Test %

Citation

@misc{sotoudehsharifi2024pathowave,
  title={PathoWAve: A Deep Learning-based Weight Averaging Method for Improving Domain Generalization in Histopathology Images},
  author={Parastoo Sotoudeh Sharifi, M. Omair Ahmad, M.N.S. Swamy (2024)},
  year={2024},
  note={arXiv:2406.15685}
}

• arXiv: 2406.15685