Multiverse Eval

multiverse-eval

MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models — Young-Jun Lee et al. (2025) (arXiv:2510.16641, 2025)

What this evaluates

Evaluates the multi-turn conversational reasoning and sustained dialogue capabilities of Vision-Language Models (VLMs) across diverse domains like mathematics, coding, and creative tasks. It probes how well models leverage dialogue history (in-context learning) and maintain consistency over extended interactions.

Datasets

• MultiVerse — total 647; splits: test (647)

Metrics

• checklist-based evaluation (primary) — range: percent
  • A percentage score derived from a GPT-4o-based checklist covering 37 aspects (e.g., perceptual accuracy, factual correctness). The model's response is evaluated against predefined criteria for each turn, and the final metric is the average percentage of aspects satisfied across all dialogues.

Input / output format

Input: A sequence of multi-turn dialogues containing images and text prompts. Each instance provides either ground-truth dialogue history (Oracle setting) or the model's own generated history (Self-Prediction setting) up to the current turn, followed by the current turn's image and user query.

Output: A natural language text response generated by the VLM for the current turn's query.

Scoring recipe

def score_dialogue(dialogue_history, current_image, current_query, model, gpt4o):
    response = model.generate(dialogue_history, current_image, current_query)
    aspects_met = 0
    for aspect in checklist_37_aspects:
        if gpt4o.evaluate(response, aspect) == 'pass':
            aspects_met += 1
    return (aspects_met / 37) * 100
# Final metric = mean(score_dialogue(d) for d in MultiVerse_test_set)

Common pitfalls

• Performance heavily depends on whether ground-truth dialogue history is provided (Oracle) or self-generated (Self-Prediction), with gaps up to ~45%.
• The evaluation metric may exhibit verbosity bias, though the paper claims mitigation; longer responses do not always correlate with higher scores.
• Models show highly uneven performance across interaction goals (e.g., strong in analysis/understanding but weak in optimization/research), so reporting a single aggregate score masks domain-specific failures.

Evidence (verbatim from paper)

employs a checklist-based evaluation using GPT-4o to assess 37 aspects including perceptual accuracy and factual correctness. ... all VLMs exhibit relatively low performance ($<50%$), indicating that multi-turn interactions in MultiVerse remain challenging, even for high-performing VLMs in the Oracle setting.

Citation

@misc{lee2025multiverse,
  title={MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models},
  author={Young-Jun Lee et al. (2025)},
  year={2025},
  note={arXiv:2510.16641}
}

• arXiv: 2510.16641