Imagenetvc Eval

imagenetvc-eval

ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories — Heming Xia et al. (2023) (arXiv:2305.15028, 2023)

What this evaluates

Evaluates zero- and few-shot visual commonsense reasoning capabilities of language models and visually-augmented language models across 1,000 ImageNet categories using human-annotated QA pairs.

Datasets

• ImageNetVC — total 1000; splits: test (-1); repo https://github.com/hemingkx/ImageNetVC

Metrics

• Top-1 accuracy (primary) — range: percent
  • Fraction of correct predictions. Model scores each candidate answer y as 1 / -log P(y|context), applies softmax over all candidates to get probabilities, and selects the argmax. Accuracy is the percentage of instances where the predicted answer matches the ground truth.

Input / output format

Input: For LLMs: text prompt formatted as '[Question] The answer is [Answer].' For VaLMs: same text prompt concatenated with top-K (K=10) images retrieved via Google Image Search or synthesized via Stable Diffusion, sorted by CLIP similarity to the question.

Output: Probability distribution over all candidate answers via softmax. Final prediction is the argmax of this distribution.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

# Per-instance prediction:
# log_probs = {y: -math.log(model.log_prob(y, context)) for y in candidates}
# probs = softmax(log_probs)
# pred = argmax(probs)

Common pitfalls

• Prompt format bias can skew raw log-probabilities; the paper uses calibration/normalization to mitigate this.
• Performance variance is high across different prompt templates; results should be averaged over multiple prompts.
• VaLM evaluation depends heavily on image retrieval/synthesis quality and CLIP-based sorting, which may not perfectly align with the question.

Evidence (verbatim from paper)

Following Schick and Schütze (2021) and Yang et al. (2022), we treat the zero-shot evaluation as a cloze test, transforming the QA pairs in ImageNetVC into prompts like “[Question] The answer is [Answer].” ... The numbers along the radio axis denote the mean Top-1 accuracy (%) of models over 5 different prompts.

Citation

@misc{xia2023imagenetvc,
  title={ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories},
  author={Heming Xia et al. (2023)},
  year={2023},
  note={arXiv:2305.15028}
}

• arXiv: 2305.15028