vlm-subtlebench-eval
VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? — Kim et al. (2026) (arXiv:2603.07888, 2026)
What this evaluates
This benchmark evaluates vision-language models' ability to perform subtle comparative reasoning between pairs of images. It probes capabilities across ten fine-grained difference types, including spatial, temporal, viewpoint, attribute, and existence changes, requiring models to detect and explain nuanced visual discrepancies that are often missed by standard prompting or simple image concatenation.
Datasets
Metrics
accuracy (primary) — range: percent
- Proportion of correct answers on multiple-choice questions, calculated as the number of matches between predicted and ground-truth options divided by the total number of questions.
cosine similarity score (CSS) — range: [0, 1]
- Cosine similarity between sentence embeddings of the generated caption and the ground-truth caption, computed using Sentence-BERT.
LLM-as-a-judge — range: percent
- Score assigned by an LLM judge evaluating the quality and relevance of generated captions against ground truth.
Input / output format
Input: A pair of images and a multiple-choice question or a prompt requesting a caption describing the differences.
Output: A selected multiple-choice option (A/B/C/D) or a free-text caption describing the subtle differences between the two images.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return correct / len(gold) * 100
Common pitfalls
- Concatenating two images horizontally into a single input degrades accuracy in 9 out of 10 domains compared to standard side-by-side or separate input formats.
- Two-step reasoning prompting can cause models to hallucinate 'no difference' during the intermediate description stage, leading to incorrect final predictions.
- Model performance is highly sensitive to object size, count, and scene complexity, not just the semantic type of difference.
Evidence (verbatim from paper)
For multiple-choice questions, performance is measured by accuracy, capturing the proportion of correct answers. For the captioning task, we apply cosine similarity score (CSS) and LLM-as-a-judge, to assess the quality and relevance of generated captions.
Citation
@misc{kim2026vlmsubtlebench,
title={VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?},
author={Kim et al. (2026)},
year={2026},
note={arXiv:2603.07888}
}
1---2name: vlm-subtlebench-eval3description: This benchmark evaluates vision-language models' ability to perform subtle comparative reasoning between pairs of images. It probes capabilities across ten fine-grained difference types, including spatial, temporal, viewpoint, attribute, and existence changes, requiring models to detect and explain nuanced visual discrepancies that are often missed by standard prompting or simple image concatenation. Use when the user wants to benchmark on VLM-SubtleBench, or asks about evaluating this task. Reports accuracy.4---56# vlm-subtlebench-eval78> VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? — Kim et al. (2026) (arXiv:2603.07888, 2026)910## What this evaluates1112This benchmark evaluates vision-language models' ability to perform subtle comparative reasoning between pairs of images. It probes capabilities across ten fine-grained difference types, including spatial, temporal, viewpoint, attribute, and existence changes, requiring models to detect and explain nuanced visual discrepancies that are often missed by standard prompting or simple image concatenation.1314## Datasets1516- **VLM-SubtleBench** — total 13000; splits: test (-1), val (-1); repo https://github.com/krafton-ai/VLM-SubtleBench1718## Metrics1920- `accuracy` **(primary)** — range: percent21 - Proportion of correct answers on multiple-choice questions, calculated as the number of matches between predicted and ground-truth options divided by the total number of questions.22- `cosine similarity score (CSS)` — range: [0, 1]23 - Cosine similarity between sentence embeddings of the generated caption and the ground-truth caption, computed using Sentence-BERT.24- `LLM-as-a-judge` — range: percent25 - Score assigned by an LLM judge evaluating the quality and relevance of generated captions against ground truth.2627## Input / output format2829**Input**: A pair of images and a multiple-choice question or a prompt requesting a caption describing the differences.3031**Output**: A selected multiple-choice option (A/B/C/D) or a free-text caption describing the subtle differences between the two images.3233## Scoring recipe3435```python36def compute_accuracy(predictions, gold):37 correct = sum(1 for p, g in zip(predictions, gold) if p == g)38 return correct / len(gold) * 10039```4041## Common pitfalls4243- Concatenating two images horizontally into a single input degrades accuracy in 9 out of 10 domains compared to standard side-by-side or separate input formats.44- Two-step reasoning prompting can cause models to hallucinate 'no difference' during the intermediate description stage, leading to incorrect final predictions.45- Model performance is highly sensitive to object size, count, and scene complexity, not just the semantic type of difference.4647## Evidence (verbatim from paper)4849> For multiple-choice questions, performance is measured by accuracy, capturing the proportion of correct answers. For the captioning task, we apply cosine similarity score (CSS) and LLM-as-a-judge, to assess the quality and relevance of generated captions.5051## Citation5253```bibtex54@misc{kim2026vlmsubtlebench,55 title={VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?},56 author={Kim et al. (2026)},57 year={2026},58 note={arXiv:2603.07888}59}60```6162- arXiv: 2603.07888
qhjqhj00