Imagenet Top1 Error Eval

imagenet-top1-error-eval

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — Goyal et al. (2017) (arXiv:1706.02677, 2017)

What this evaluates

Evaluates the top-1 classification accuracy of a model on the ImageNet dataset. It probes the model's ability to correctly classify images into one of 1000 categories under various training conditions, specifically testing the impact of large minibatch sizes and learning rate scaling strategies on optimization and generalization.

Datasets

• ImageNet — total 1330000; splits: train (1280000), val (50000); HF imagenet-1k

Metrics

• top-1 error (%) (primary) — range: percent
  • 100 * (1 - (number of correctly classified validation images / total number of validation images)).

Input / output format

Input: 224x224 pixel random crops from augmented images (with horizontal flips), normalized by per-color mean and standard deviation.

Output: Class label prediction (top-1) for each validation image.

Scoring recipe

correct = 0
for img, label in val_dataset:
    pred = model(img)
    if pred == label:
        correct += 1
error_rate = 100 * (1 - correct / len(val_dataset))

Common pitfalls

• Error is computed as the median over the final 5 training epochs, not just the final epoch.
• Results must be averaged over 5 independent runs to account for random variation; single-run results are unreliable.

Evidence (verbatim from paper)

The 1000-way ImageNet classification task serves as our main experimental benchmark. Models are trained on the ~1.28 million training images and evaluated by top-1 error on the 50,000 validation images. ... we compute a model’s error rate as the median error of the final 5 epochs. Moreover, we report the mean and standard deviation (std) of the error from 5 independent runs.

Citation

@misc{goyal2017accurate,
  title={Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour},
  author={Goyal et al. (2017)},
  year={2017},
  note={arXiv:1706.02677}
}

• arXiv: 1706.02677