Mmdr Bench Eval

mmdr-bench-eval

ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following — Han et al. (2025) (arXiv:2508.15164, 2025)

What this evaluates

This benchmark evaluates a model's ability to handle complex, multi-turn visually-grounded dialogue and follow intricate instructions. It probes sustained contextual understanding, visual entity tracking across turns, and multi-step reasoning depth in dynamic multi-modal interactions.

Datasets

• MMDR-Bench — total 300; splits: test (300)

Metrics

• average human evaluation ratings (primary) — range: [1, 5]
  • Human experts score each model response per turn on a 1-5 scale across six dimensions: visual entity tracking, dialogue consistency, reasoning depth, instruction adherence, error suppression, and response fluency. The final metric is the arithmetic mean of all turn-level scores across all scenarios and evaluators.

Input / output format

Input: Multi-turn dialogue scenarios (5-7 turns on average) centered around one or more images, containing complex question-answering or instruction-following prompts per turn.

Output: Textual model responses for each dialogue turn.

Scoring recipe

scores = []
for scenario in dataset:
    for turn in scenario.turns:
        rating = human_rater.evaluate(turn.prompt, turn.image, turn.model_response, dimensions=6)
        scores.append(rating)
return sum(scores) / len(scores)

Common pitfalls

• Human evaluation relies on expert raters, introducing potential subjectivity and inter-rater variability that automated metrics cannot capture.
• LLM-based automatic evaluation is mentioned only as supplementary validation without a defined formula or prompt, making it non-reproducible.
• Latency measurements are hardware-dependent and may vary significantly across different deployment environments.

Evidence (verbatim from paper)

Table[I] presents the core experimental results, showcasing the average human evaluation ratings of different models across the six critical dimensions on the MMDR-Bench dataset. Human evaluators, who are experts in multi-modal AI interactions, meticulously review model responses for each dialogue turn and assign a score on a 1-5 scale across the six predefined dimensions: visual entity tracking, dialogue consistency, reasoning depth, instruction adherence, error suppression, and response fluency. A score of 5 indicates excellent performance, while 1 indicates poor performance. The final reported scores are the average ratings across all scenarios and evaluators.

Citation

@misc{han2025contextualvlmagent,
  title={ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following},
  author={Han et al. (2025)},
  year={2025},
  note={arXiv:2508.15164}
}

• arXiv: 2508.15164