# Cinic 10 Eval

> Evaluates image classification performance under significant domain shift between synthetic (CIFAR-10) and real-world/downsampled (ImageNet) sources. Probes model robustness to distributional bias and class-level statistical divergence across training and test domains. Use when the user wants to benchmark on CINIC-10, or asks about evaluating this task. Reports Test Error.

- Skill: `qhjqhj00/cinic-10-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/cinic-10-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/cinic-10-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/cinic-10-eval

---


# cinic-10-eval

> CINIC-10 is not ImageNet or CIFAR-10 — Darlow et al. (2018) (arXiv:1810.03505, 2018)

## What this evaluates

Evaluates image classification performance under significant domain shift between synthetic (CIFAR-10) and real-world/downsampled (ImageNet) sources. Probes model robustness to distributional bias and class-level statistical divergence across training and test domains.

## Datasets

- **CINIC-10** — total 270000; splits: train (-1), val (-1), test (-1); repo https://github.com/BayesWatch/cinic-10

## Metrics

- `Test Error` **(primary)** — range: percent
  - Computed as 1 minus top-1 accuracy. The paper reports the mean and standard deviation across multiple experimental runs.

## Input / output format

**Input**: 32×32 RGB images with corresponding class labels from 10 categories.

**Output**: Predicted class label (or class probabilities) for each image.

## Scoring recipe

```python
def compute_test_error(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    return (1 - accuracy) * 100  # Returns percentage
```

## Common pitfalls

- Models may exploit synthetic artifacts present in the downsampled ImageNet images rather than learning robust features.
- The domain shift between CIFAR-10 and ImageNet causes distributional bias that standard data augmentation may not fully mitigate.
- Reporting only mean test error without confidence intervals or multiple seeds can mask high variance in domain adaptation performance.

## Evidence (verbatim from paper)

> Table 1: CINIC-10 benchmarks.  

<table><tr><td>Model</td><td>No. Parameters</td><td>Test Error</td></tr>...

## Citation

```bibtex
@misc{darlow2018cinic10,
  title={CINIC-10 is not ImageNet or CIFAR-10},
  author={Darlow et al. (2018)},
  year={2018},
  note={arXiv:1810.03505}
}
```

- arXiv: 1810.03505