qhjqhj00 cross-domain-meta-dl-eval
NeurIPS'22 Cross-Domain MetaDL competition: Design and baseline results — Carrión-Ojeda et al. (2022) (arXiv:2208.14686, 2022)
What this evaluates
Probes few-shot image classification generalization across diverse domains and highly variable task regimes (2–20 ways, 1–20 shots) without relying on pre-trained backbones.
Datasets
- Meta-Album — total ?; splits: test (-1); repo https://github.com/DustinCarrion/cd-metacl
Metrics
accuracy(primary) — range: percent- Percentage of correctly classified query images in each few-shot task, averaged across all tasks.
Input / output format
Input: Image classification tasks with variable numbers of ways (2–20) and shots (1–20) drawn from 10 diverse domains.
Output: Class predictions for query images in each task.
Scoring recipe
correct = 0
total = 0
for task in tasks:
preds = model.predict(task.query_images)
correct += sum(p == g for p, g in zip(preds, task.ground_truth))
total += len(task.query_images)
return correct / total
Common pitfalls
- Using pre-trained backbones is explicitly forbidden; models must be trained de novo.
- Task regimes are not fixed; models must handle variable ways (2–20) and shots (1–20) in a single evaluation.
- Cross-domain generalization is required; performance on seen domains does not guarantee performance on unseen ones.
Evidence (verbatim from paper)
Introduces a novel cross-domain meta-learning challenge (NeurIPS'22 Cross-Domain MetaDL) that evaluates models on 'any-way, any-shot' few-shot image classification across 10 diverse domains (e.g., healthcare, ecology), using Meta-Album—a curated meta-dataset of 40 image classification datasets. Unlike prior benchmarks, it emphasizes domain generalization, variable task regimes (2–20 ways, 1–20 shots), and requires 'de novo' training without pre-trained backbones, testing true meta-learning generalization.
Citation
@misc{carrionojeda2022crossdomainmetadl,
title={NeurIPS'22 Cross-Domain MetaDL competition: Design and baseline results},
author={Carrión-Ojeda et al. (2022)},
year={2022},
note={arXiv:2208.14686}
}
- arXiv: 2208.14686