Blink Vision Centric Eval

blink-vision-centric-eval

Hidden in plain sight: VLMs overlook their visual representations — Fu et al. (2025) (arXiv:2506.08008, 2025)

What this evaluates

This evaluation probes a model's ability to leverage raw visual representations for vision-centric tasks without relying on language priors or domain expertise. It tests pixel-level matching, depth perception, 3D object awareness, and art style recognition across multiple-choice and regression-style tasks.

Datasets

• CV-Bench (Depth Order) — total ?; splits: test (-1)
• SPair-71k — total ?; splits: test (-1)
• FunKPoint — total ?; splits: test (-1)
• HPatches — total ?; splits: test (-1)
• MOCHI — total ?; splits: test (-1)
• WikiArt (BLINK Art Style) — total ?; splits: test (-1)

Metrics

• multiple-choice VQA (primary) — range: [0, 1]
  • Percentage of correctly selected options in a multiple-choice VQA format. For vision encoders, task-specific metrics are used: depth comparison accuracy, cosine similarity matching, and MSE between Gram matrices for style.

Input / output format

Input: One or two images (reference/target), optionally with bounding boxes or dots, accompanied by a multiple-choice prompt asking to identify the correct match, closer object, or matching style.

Output: A single letter option (e.g., 'A', 'B', 'C', 'D') or a predicted depth/value for encoder evaluation.

Scoring recipe

def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
# For encoders: compare predicted values (depth, similarity, MSE) against ground truth thresholds or select argmax/argmin.

Common pitfalls

• Assuming strong vision encoder performance implies strong VLM performance; the LLM projection/decoding layer is often the bottleneck.
• Confusing language priors or domain knowledge with actual visual reasoning capabilities.
• Overlooking prompt sensitivity; small prompt changes can drastically alter VLM performance on vision tasks.

Evidence (verbatim from paper)

We evaluate VLMs through a multiple-choice VQA format, and evaluate the vision encoder by computing pairwise cosine similarity of CLS embeddings and choosing the example with the lowest average score.

Citation

@misc{fu2025hidden,
  title={Hidden in plain sight: VLMs overlook their visual representations},
  author={Fu et al. (2025)},
  year={2025},
  note={arXiv:2506.08008}
}

• arXiv: 2506.08008