Do You See Me Eval

do-you-see-me-eval

Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs — Kanade et al. (2025) (arXiv:2506.02022, 2025)

What this evaluates

This benchmark isolates and evaluates core visual perception capabilities in multimodal large language models (MLLMs) through programmatically generated tasks inspired by human psychology. It probes figure-ground discrimination, spatial relations, visual form constancy, and perceptual brittleness across 2D and 3D settings with controlled difficulty levels.

Datasets

• Do You See Me — total ?; splits: test (-1); repo https://github.com/microsoft/Do-You-See-Me

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions after parsing the model's extended output into a standardized label or value. Calculated as the number of correct predictions divided by the total number of samples.

Input / output format

Input: An image paired with a predefined textual prompt containing a perception question (and optionally a reasoning question for the joint dataset), instructing the model to generate a detailed chain-of-thought response.

Output: A detailed textual response containing chain-of-thought reasoning, which is subsequently parsed by an answer extractor (GPT-4o) into a concise standardized answer (e.g., multiple-choice label or numeric value).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    # predictions: list of raw model outputs
    # gold_labels: list of ground truth answers
    extracted = [extract_answer(pred) for pred in predictions]  # GPT-4o parser
    standardized = [standardize(extracted_i) for extracted_i in extracted]  # to MCQ label or number
    correct = sum(1 for pred, gold in zip(standardized, gold_labels) if pred == gold)
    return correct / len(gold_labels)

Common pitfalls

• Models can arrive at correct final answers despite misperceiving crucial visual elements (observed in 29% of correct responses).
• Chain-of-Thought prompting degrades performance by introducing lossy text representations of visual content.
• Models may exploit multiple-choice cues or prompt structure rather than performing genuine visual analysis.

Evidence (verbatim from paper)

Given that modern LLMs and MLLMs often generate extended textual outputs, making traditional rule-based or template-matching evaluation methods challenging*[[24]], we employ an expert LLM-based evaluation protocol, inspired by recent benchmarks[[25], [47]]. This framework proceeds in three stages: First, the MLLM under evaluation generates a detailed response following a predefined input template (see [Appendix J] for prompt structures). Second, an answer extractor, specifically GPT-4o [[26]] (the same version as evaluated), parses these extended outputs to yield a concise answer. This approach leverages the high accuracy of expert LLMs in answer extraction, as demonstrated in prior work[[25]]*. Third, the extracted text is standardized (e.g., reduced to multiple-choice labels or numeric values). Performance across all subtasks is measured using accuracy.

Citation

@misc{kanade2025doyouseeme,
  title={Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs},
  author={Kanade et al. (2025)},
  year={2025},
  note={arXiv:2506.02022}
}

• arXiv: 2506.02022