Mmvp Eval

mmvp-eval

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs — Tong et al. (2024) (arXiv:2401.06209, 2024)

What this evaluates

Evaluates the visual reasoning and grounding capabilities of multimodal large language models (MLLMs) on basic visual patterns such as orientation, counting, viewpoint, and feature presence. It specifically probes whether models fail due to limitations in their visual encoders (e.g., CLIP) rather than language model hallucinations.

Datasets

• MMVP — total ?; splits: (unstated)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100. The paper also reports average accuracy across visual pattern categories.
• Pearson Correlation — range: [-1, 1]
  • Measures the linear correlation between CLIP model errors and MLLM errors on the benchmark, or between ImageNet-1k zero-shot accuracy and MMVP performance.

Input / output format

Input: An image paired with a multiple-choice visual question (e.g., 'Are the butterfly's wings closer to being open or closed? (a) Open (b) Closed').

Output: The model's selected option letter or text answer corresponding to the ground truth.

Scoring recipe

correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
accuracy = (correct / len(predictions)) * 100
return accuracy

Common pitfalls

• Assuming low performance is due to language model hallucination rather than visual encoder limitations (addressed via option-swapping ablations).
• Option-order bias: models may favor certain positions (e.g., 'a' or 'b') regardless of content, requiring randomized or swapped option testing.
• Over-reliance on CLIP embeddings leads to systematic failures on 'CLIP-blind pairs' where visually distinct images have similar embeddings.

Evidence (verbatim from paper)

We collect their responses and calculate the average score as the human-level performance. ... GPT-4V obtains a 40.3% accuracy on the option swapping in our study, as opposed to the original 38.7%. ... We compute the Pearson Correlation between the CLIP model and MLLMs and show results in Table 5.

Citation

@misc{tong2024eyeswideshut,
  title={Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs},
  author={Tong et al. (2024)},
  year={2024},
  note={arXiv:2401.06209}
}

• arXiv: 2401.06209