Wilds Eval

wilds-eval

WILDS: A Benchmark of in-the-Wild Distribution Shifts — Pang Wei Koh et al. (2020) (arXiv:2012.07421, 2020)

What this evaluates

Evaluates machine learning models' robustness to real-world distribution shifts, specifically domain generalization and subpopulation shifts. It measures how much model performance degrades when tested on out-of-distribution (OOD) data compared to in-distribution (ID) data, highlighting gaps in generalization for real-world deployment.

Datasets

• WILDS — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/p-lambda/wilds

Metrics

• ID and OOD performance (primary) — range: [0, 1]
  • Reports both in-distribution (ID) and out-of-distribution (OOD) performance, typically calculated as accuracy or error rate on the respective data splits.

Input / output format

Input: Standardized dataset instances (e.g., medical images, satellite imagery, codebases) with ground-truth labels.

Output: Model predictions for the target task.

Scoring recipe

def compute_metrics(predictions, gold, id_mask):
    id_acc = (predictions[id_mask] == gold[id_mask]).mean()
    ood_acc = (predictions[~id_mask] == gold[~id_mask]).mean()
    return {'id_performance': id_acc, 'ood_performance': ood_acc}

Common pitfalls

• Overfitting to the limited OOD test domains (e.g., single hospital in CAMELYON17-wILDS)
• Failing to report both ID and OOD performance separately
• Using external data or non-default architectures that confound algorithmic contributions

Evidence (verbatim from paper)

To better understand the extent to which any gains in OOD performance can be attributed to improved ID performance versus a model that is more robust to (i.e., less affected by) the distribution shift, we encourage model developers to report both ID and OOD performance numbers.

Citation

@misc{koh2020wilds,
  title={WILDS: A Benchmark of in-the-Wild Distribution Shifts},
  author={Pang Wei Koh et al. (2020)},
  year={2020},
  note={arXiv:2012.07421}
}

• arXiv: 2012.07421