Cinic 10 Eval

cinic-10-eval

CINIC-10 is not ImageNet or CIFAR-10 — Darlow et al. (2018) (arXiv:1810.03505, 2018)

What this evaluates

Evaluates image classification performance under significant domain shift between synthetic (CIFAR-10) and real-world/downsampled (ImageNet) sources. Probes model robustness to distributional bias and class-level statistical divergence across training and test domains.

Datasets

• CINIC-10 — total 270000; splits: train (-1), val (-1), test (-1); repo https://github.com/BayesWatch/cinic-10

Metrics

• Test Error (primary) — range: percent
  • Computed as 1 minus top-1 accuracy. The paper reports the mean and standard deviation across multiple experimental runs.

Input / output format

Input: 32×32 RGB images with corresponding class labels from 10 categories.

Output: Predicted class label (or class probabilities) for each image.

Scoring recipe

def compute_test_error(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    return (1 - accuracy) * 100  # Returns percentage

Common pitfalls

• Models may exploit synthetic artifacts present in the downsampled ImageNet images rather than learning robust features.
• The domain shift between CIFAR-10 and ImageNet causes distributional bias that standard data augmentation may not fully mitigate.
• Reporting only mean test error without confidence intervals or multiple seeds can mask high variance in domain adaptation performance.

Evidence (verbatim from paper)

Table 1: CINIC-10 benchmarks.

Citation

@misc{darlow2018cinic10,
  title={CINIC-10 is not ImageNet or CIFAR-10},
  author={Darlow et al. (2018)},
  year={2018},
  note={arXiv:1810.03505}
}

• arXiv: 1810.03505