Meta Omnium Eval

meta-omnium-eval

Meta Omnium: A Benchmark for General-Purpose Learning-to-Learn — Bohdal et al. (2023) (arXiv:2305.07625, 2023)

What this evaluates

Evaluates few-shot meta-learners and transfer learning baselines on their ability to generalize across heterogeneous vision tasks including classification, semantic segmentation, keypoint localization, and regression. It specifically probes cross-task knowledge transfer, in-distribution versus out-of-distribution robustness, and the comparative effectiveness of single-task versus multi-task meta-training protocols.

Datasets

• Meta Omnium — total ?; splits: meta-training (-1), meta-validation (-1), meta-testing (-1)

Metrics

• accuracy — range: percent
  • Percentage of correctly predicted class labels on query images within few-shot episodes.
• mIOU — range: percent
  • Mean Intersection over Union computed across all semantic classes for predicted segmentation masks.
• PCK — range: percent
  • Percentage of Correct Keypoints, measuring localization accuracy where predicted coordinates fall within a normalized distance threshold of the ground truth.
• Average Rank (primary) — range: other
  • Mean rank of a method across all datasets within a task type (and across task types), computed by ordering methods by performance on each dataset and averaging the resulting ranks.

Input / output format

Input: Few-shot learning episodes consisting of support sets (images with labels, masks, or keypoint coordinates) and query sets for classification, segmentation, keypoint localization, or regression tasks.

Output: Predictions for query set items: class labels, pixel-wise segmentation masks, keypoint coordinates, or continuous regression values.

Scoring recipe

def compute_scores(predictions, gold, task_type):
    if task_type == 'classification':
        return accuracy(predictions, gold)
    elif task_type == 'segmentation':
        return mIOU(predictions, gold)
    elif task_type == 'keypoints':
        return PCK(predictions, gold)
    # For regression, use dataset-specific metric
    return dataset_metric(predictions, gold)

# Aggregate across 600 meta-testing tasks per dataset
# Average scores across datasets within each task type
# Compute Average Rank by sorting methods per dataset and averaging ranks

Common pitfalls

• Evaluating only in-distribution (ID) tasks masks significant performance degradation on out-of-distribution (OOD) datasets, particularly for segmentation and keypoint tasks.
• Assuming multi-task meta-training automatically outperforms single-task training ignores the negative impact of representation heterogeneity across task families.
• Applying standard ImageNet pre-training without conditional evaluation may yield inconsistent or negligible gains in this multi-task meta-learning setting.

Evidence (verbatim from paper)

Classification, segmentation, and keypoint results are reported in accuracy (%), mIOU (%), and PCK (%) respectively. The table also reports the average rank of each meta-learner across each dataset, both overall and broken down by ID and OOD datasets.

Citation

@misc{bohdal2023metaomnium,
  title={Meta Omnium: A Benchmark for General-Purpose Learning-to-Learn},
  author={Bohdal et al. (2023)},
  year={2023},
  note={arXiv:2305.07625}
}

• arXiv: 2305.07625