Vl Rethinker Eval

vl-rethinker-eval

VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning — Wang et al. (2025) (arXiv:2504.08837, 2025)

What this evaluates

Evaluates the multimodal reasoning and self-reflection capabilities of vision-language models across math, multi-discipline, and real-world benchmarks. It probes whether models can correctly interpret visual-textual inputs and produce accurate final answers under greedy decoding.

Datasets

• MathVista — total ?; splits: testmini (-1)
• MathVerse — total ?; splits: testmini (-1)
• MathVision — total ?; splits: test (-1)
• MMMU-Pro — total ?; splits: overall (-1)
• MMMU — total ?; splits: val (-1), full (-1)
• EMMA — total ?; splits: full (-1)
• MegaBench — total ?; splits: core (-1)

Metrics

• Pass@1 accuracy (primary) — range: percent
  • Calculated as the fraction of queries where the model's single greedy-decoded response exactly matches the ground truth answer.

Input / output format

Input: Multimodal queries consisting of an image and a text prompt/question.

Output: A single final answer string generated via greedy decoding.

Scoring recipe

correct = 0
for query, gold in dataset:
    pred = model.generate(query, decoding='greedy')
    if normalize(pred) == normalize(gold):
        correct += 1
return (correct / len(dataset)) * 100

Common pitfalls

• Using sampling or beam search instead of greedy decoding, which violates the specified evaluation protocol.
• Mixing up benchmark splits (e.g., using MathVista test instead of testmini, or MMMU full instead of val), as results are split-specific.
• Ignoring the multimodal nature of the input; the protocol strictly requires image-text pairs, not text-only queries.

Evidence (verbatim from paper)

For evaluation, we employ a diverse set of challenging multimodal benchmarks: MathVista, MathVerse, and MathVision. Multi-discipline understanding and reasoning: MMMU, MMMU-Pro, and EMMA. Large-scale long-tailed real-world tasks: MegaBench. This benchmark suite covers a wide range of complex multimodal reasoning challenges. We report the Pass@1 accuracy using greedy decoding.

Citation

@misc{wang2025vlrethinker,
  title={VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2504.08837}
}

• arXiv: 2504.08837