Mmstar Eval

mmstar-eval

Are We on the Right Way for Evaluating Large Vision-Language Models? — Lin Chen et al. (2024) (arXiv:2403.20330, 2024)

What this evaluates

Evaluates Large Vision-Language Models (LVLMs) on six core capabilities (coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and mathematics) using a human-curated benchmark designed to enforce strict visual dependency and minimize data leakage.

Datasets

• MMStar — total 1500; splits: test (1500)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions, determined via heuristic matching rules against the gold option.
• multi-modal gain (MG) — range: percent
  • Measures the actual performance improvement attributable to vision, calculated as the difference between accuracy with image and accuracy without image.
• multi-modal leakage (ML) — range: percent
  • Quantifies data leakage by measuring model accuracy when evaluated in a text-only setting (LVLM-text), indicating reliance on memorized text rather than visual reasoning.

Input / output format

Input: Multiple-choice question paired with an image (for visual evaluation) or text-only (for leakage evaluation). Models receive the question and options.

Output: Selected option letter/text from the multiple-choice answers.

Scoring recipe

# Per instance
pred = heuristic_match(model_output, options)
acc = 1.0 if pred == gold else 0.0

# Aggregation
acc_with_img = mean(acc_with_image)
acc_without_img = mean(acc_without_image) # LVLM-text setting
MG = acc_with_img - acc_without_img
ML = acc_without_img

Common pitfalls

• LLMs often refuse to answer visual questions without images; using 2-shot prompting for LLMs on MMStar is required to prevent refusal bias from skewing results.
• Some LVLMs (e.g., GeminiPro-Vision, CogVLM) crash or behave incorrectly if image tokens are simply removed; they require replacing images with pure grey images to properly evaluate the text-only setting.
• Questions must be strictly visually dependent; answers derivable from text alone or world knowledge invalidate the visual dependency control.

Evidence (verbatim from paper)

In this section, we present the results of our proposed multi-modal gain (MG) and multi-modal leakage (ML) metrics of 16 LVLMs with varying sizes and architectures on 6 popular benchmarks and our MMStar benchmark. Given that all questions are ensured to be converted into a multiple-choice format, we develop some heuristic matching rules to calculate accuracy, avoiding the cumbersome process of re-invoking GPT4 for answer extraction.

Citation

@misc{chen2024mmstar,
  title={Are We on the Right Way for Evaluating Large Vision-Language Models?},
  author={Lin Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.20330}
}

• arXiv: 2403.20330