Fine Grained Classification Eval

fine-grained-classification-eval

Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models — Ghosh et al. (2026) (arXiv:2602.17871, 2026)

What this evaluates

Probes the ability of vision-language models to distinguish visually similar subcategories within broader classes (e.g., specific bird species or car models) using a multiple-choice format.

Datasets

• ImageNet-1K — total ?; splits: test (-1)
• Oxford Flowers-102 — total ?; splits: test (-1)
• Oxford-IIIT Pet-37 — total ?; splits: test (-1)
• Food-101 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match accuracy: the proportion of test images where the model's predicted option exactly matches the ground-truth class label.

Input / output format

Input: An image paired with five multiple-choice options (A–E), where one option is the ground-truth class and the other four are hard negatives selected via OpenCLIP ViT-L/14 cosine similarity.

Output: A single letter (A, B, C, D, or E) corresponding to the selected class option.

Scoring recipe

correct = 0
total = 0
for img, options, ground_truth in dataset:
    pred = model.generate(img, options)
    if pred == ground_truth:
        correct += 1
    total += 1
accuracy = correct / total

Common pitfalls

• The multiple-choice options are dynamically generated per image using OpenCLIP cosine similarity rather than being fixed across the dataset.
• Performance is measured on exact match to the option letter, not semantic similarity or free-text generation.

Evidence (verbatim from paper)

We evaluate the fine-grained visual knowledge of VLMs using four well-established object recognition benchmarks, each focusing on different domains of fine-grained classification: ImageNet-1K, Oxford Flowers-102, Oxford-IIIT Pet-37, and Food-101. For each VLM, we use the respective default prompt format from VLMEvalKit and measure accuracy based on the exact match with the multiple choice options.

Citation

@misc{ghosh2026finegrained,
  title={Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models},
  author={Ghosh et al. (2026)},
  year={2026},
  note={arXiv:2602.17871}
}

• arXiv: 2602.17871