Imagenet Zoom Classification Eval

imagenet-zoom-classification-eval

ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification — Taesiri et al. (2023) (arXiv:2304.05538, 2023)

What this evaluates

Evaluates image classification models' accuracy on standard and out-of-distribution datasets. It specifically probes the impact of spatial zooming and foreground/background signal separation on model performance, revealing how much background cues contribute to classification accuracy.

Datasets

• ImageNet — total ?; splits: test (-1)
• ImageNet-A — total ?; splits: test (-1)
• ObjectNet — total ?; splits: test (-1)

Metrics

• top-1 accuracy (primary) — range: percent
  • Percentage of correctly predicted class labels out of the total number of test instances.
• Mann-Whitney U test p-value — range: [0, 1]
  • Non-parametric statistical test used to determine if there is a significant difference in the distribution of object counts between two datasets.

Input / output format

Input: RGB images (original resolution or resized) fed into a pretrained vision classifier.

Output: Class label predictions (top-1) or aggregated accuracy scores across multiple zoom crops.

Scoring recipe

def compute_top1_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models heavily rely on background cues; masking background (BGSet) drastically drops accuracy, so standard ImageNet evaluation can overestimate robustness to spatial bias.
• Zoom-based test-time augmentation requires searching over multiple crop scales/positions; using a single 1-crop baseline significantly underestimates the model's potential accuracy.

Evidence (verbatim from paper)

Table A8: ImageNet classification from object-only and background-only signals. Numbers show the maximum possible top-1 accuracy (%) using zoom-based transforms for minimum set covers in Appendix B.4. We discover that background signals potentially hold significance for image classification.

Citation

@misc{taesiri2023imagenethard,
  title={ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification},
  author={Taesiri et al. (2023)},
  year={2023},
  note={arXiv:2304.05538}
}

• arXiv: 2304.05538