Omniview Zero Shot Eval

omniview-zero-shot-eval

Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models — Shouwei Ruan et al. (2024) (arXiv:2404.12139, 2024)

What this evaluates

This evaluation probes the viewpoint invariance and robustness of vision-language pre-training models. It measures how well models maintain classification accuracy on clean data, common out-of-distribution shifts, and specifically challenging viewpoint-variant images compared to standard baselines.

Datasets

• ImageNet-1K — total ?; splits: test (-1)
• ImageNet-V+ — total ?; splits: test (-1)
• ImageNet-V — total ?; splits: test (-1)
• OOD-CV — total ?; splits: test (-1)
• MIRO — total ?; splits: test (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Calculated as the number of correctly predicted samples divided by the total number of samples in the benchmark, multiplied by 100.

Input / output format

Input: Single RGB image per instance.

Output: Predicted class label (and confidence scores).

Scoring recipe

correct = 0
for img, gold_label in dataset:
    logits = model.encode(img)
    pred_label = argmax(logits)
    if pred_label == gold_label:
        correct += 1
top1_acc = (correct / len(dataset)) * 100

Common pitfalls

• Confusing viewpoint-OOD benchmarks (e.g., ImageNet-V, MIRO) with standard distribution-shift benchmarks (e.g., ImageNet-V2, ImageNet-Sketch), which measure different robustness properties.
• Failing to follow the standard CLIP prompting and candidate category name conventions, which significantly impacts zero-shot classification performance.
• Reporting average accuracy across all benchmarks without separating clean, common-OOD, and viewpoint-OOD domains, obscuring the model's specific viewpoint invariance gains.

Evidence (verbatim from paper)

Datasets and Metrics. We employ a various set of benchmarks for evaluation, including clean data distributions (ImageNet [13] and CIFAR [28]), common 2D-OOD (ImageNet-V2 [43], ImageNet-Sketch [56], ImageNet-O [22], ImageNet-R [20] and OOD-CV [60]), and most importantly, viewpoint-OOD (ImageNet-V [15], ImageNet-V+ [47], OOD-CV(Pose) [60] and MIRO [7]) datasets. For each benchmark, we report Top-1 and Top-5 accuracy and average accuracy across all benchmarks. The evaluations follow the standard prompting engineering and candidate category names conventions of CLIP [41].

Citation

@misc{ruan2024omniviewtuning,
  title={Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models},
  author={Shouwei Ruan et al. (2024)},
  year={2024},
  note={arXiv:2404.12139}
}

• arXiv: 2404.12139