Multimodal Reasoning Eval

multimodal-reasoning-eval

VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding — Ding et al. (2025) (arXiv:2509.24776, 2025)

What this evaluates

This evaluation probes the ability of multimodal large language models to perform complex reasoning across diverse domains (mathematics, science, diagram comprehension, and creative tasks) by requiring them to explicitly ground their reasoning in visual and textual evidence before producing a final answer.

Datasets

• MMMU — total 11500; splits: test (-1)
• MathVista — total 6141; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• EMMA — total ?; splits: test (-1)
• Creation-MMBench — total 765; splits: test (-1)
• Creation-MMBench-TO — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard benchmark accuracy calculated as the percentage of correctly answered questions out of the total number of questions in each benchmark.

Input / output format

Input: Image-text pairs containing a visual input and a question/prompt requiring reasoning.

Output: Structured text containing three distinct sections: (task-relevant perceptual evidence), (reasoning chain), and (final response).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        # Extract answer from structured output
        ans = pred.split('<answer>')[-1].strip().rstrip('</answer>')
        if ans == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Confusing the training datasets (LLaVA-CoT, Vision-SR1, MMK12, etc.) with the evaluation benchmarks.
• Failing to parse the structured , , format correctly when extracting the final answer for scoring.
• Applying uniform scoring across benchmarks without accounting for dataset-specific formats (e.g., multiple-choice vs. open-ended).

Evidence (verbatim from paper)

The reward function combined answer accuracy, format compliance, key visual/textual information, n-gram penalty, and consistency, with tuned weights.

Citation

@misc{ding2025vtpereptionr1,
  title={VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding},
  author={Ding et al. (2025)},
  year={2025},
  note={arXiv:2509.24776}
}

• arXiv: 2509.24776