Mm Vet V2 Eval

mm-vet-v2-eval

MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities — Weihao Yu et al. (arXiv:2408.00765, 2024)

What this evaluates

This benchmark evaluates large multimodal models on integrated vision-language capabilities, with a specific focus on sequential image-text understanding, spatial reasoning, knowledge retrieval, and long-form generation. It probes how well models can process interleaved visual and textual inputs to answer complex, real-world questions.

Datasets

• MM-Vet v2 — total 517; splits: test (517); repo https://github.com/yuweihao/MM-Vet

Metrics

• MM-Vet-v2 score (primary) — range: percent
  • An LLM-as-a-judge metric where GPT-4 scores model outputs on a 0–1 scale based on a prompt containing the question, ground truth, and model prediction. The final score is the average of 5 independent GPT-4 evaluations, reported as a percentage out of 100.

Input / output format

Input: Interleaved image-text sequences paired with open-ended questions or prompts requiring multimodal reasoning.

Output: Free-form text responses generated by the LMM to answer the provided questions.

Scoring recipe

scores = []
for _ in range(5):
    judge_prompt = build_prompt(question, ground_truth, model_output)
    raw_score = gpt4_judge(judge_prompt) # returns float in [0, 1]
    scores.append(raw_score)
final_score = (sum(scores) / len(scores)) * 100

Common pitfalls

• LLM-as-a-judge scoring exhibits inherent variance even at temperature 0, necessitating multiple runs (5) and reporting of standard deviation.
• The judge prompt includes the ground truth, which may bias GPT-4 toward leniency or specific phrasing rather than evaluating semantic equivalence.
• Sequential image-text understanding requires models to handle interleaved modalities, which standard single-image VLMs often fail at without explicit architectural support.

Evidence (verbatim from paper)

As illustrated in [1], for each sample, we complete the prompt template with its question, ground truth, and the output from a specific LMM. When this filled prompt is input into GPT-4, it generates a score ranging from 0 to 1 for each sample. Although the temperature is set to 0, we observe some variance in GPT-4’s outputs. To address this, we evaluate the outputs of the LLMs using GPT-4 five times. Due to space constraints, we present the average scores for capabilities and for some capability integrations, and include both the average and variance for the total score.

Citation

@misc{yu2024mmvetv2,
  title={MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities},
  author={Weihao Yu et al.},
  year={2024},
  note={arXiv:2408.00765}
}

• arXiv: 2408.00765