Domain Generalization Eval

domain-generalization-eval

Rethinking the Evaluation Protocol of Domain Generalization — Han Yu et al. (2023) (arXiv:2305.15253, 2023)

What this evaluates

This benchmark evaluates a model's out-of-distribution (OOD) generalization capability across multiple domain-shift datasets. It specifically probes whether models rely on true domain-invariant features learned from training domains versus leaking test-domain information through ImageNet pretraining weights or oracle hyperparameter selection. The protocol mandates training from scratch without pretrained weights and evaluating across multiple test domains to ensure a fair comparison of OOD generalization algorithms.

Datasets

• PACS — total 9991; splits: train (-1), val (-1), test (-1)
• VLCS — total 10729; splits: train (-1), val (-1), test (-1)
• OfficeHome — total 15588; splits: train (-1), val (-1), test (-1)
• DomainNet — total 586575; splits: train (-1), val (-1), test (-1)
• NICO++ — total 88866; splits: train (-1), val (-1), test (-1)

Metrics

• test accuracy (primary) — range: percent
  • Average classification accuracy across all test domains, computed as the mean of per-domain accuracy scores. Reported as percentage ± standard deviation over random seeds.

Input / output format

Input: RGB images from multiple source domains with corresponding class labels.

Output: Class predictions (or probability distributions) for each test image.

Scoring recipe

def compute_accuracy(preds, golds):
    return sum(p == g for p, g in zip(preds, golds)) / len(golds) * 100

def evaluate(model, test_domains):
    domain_accs = []
    for domain in test_domains:
        preds = model.predict(domain.images)
        domain_accs.append(compute_accuracy(preds, domain.labels))
    return sum(domain_accs) / len(domain_accs)

Common pitfalls

• Using ImageNet-pretrained weights for initialization, which can leak test-domain information for datasets containing real photos and inflate OOD performance.
• Performing oracle model selection by tuning hyperparameters on the test set instead of a validation set drawn from the training distribution.
• Evaluating on only a single test domain per model, which makes it easier to overfit hyperparameters to that specific domain compared to evaluating across multiple test domains simultaneously.

Evidence (verbatim from paper)

For IID model selection, we choose the test accuracy corresponding to the hyperparameters with the highest accuracy on validation data. For oracle model selection, we directly choose the highest test accuracy across all hyperparameter sets.

Citation

@misc{yu2023rethinking,
  title={Rethinking the Evaluation Protocol of Domain Generalization},
  author={Han Yu et al. (2023)},
  year={2023},
  note={arXiv:2305.15253}
}

• arXiv: 2305.15253