Mme Realworld Vbench Eval

mme-realworld-vbench-eval

High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning — Xinyu Huang et al. (2025) (arXiv:2507.05920, 2025)

What this evaluates

This evaluation probes a model's ability to perform high-resolution visual reasoning and fine-grained grounding on complex, real-world images. It specifically tests whether the model can accurately localize relevant visual regions and correctly answer multiple-choice questions without explicit grounding supervision.

Datasets

• MME-Realworld — total 23609; splits: train (21690), test (1919)
• V Bench* — total 191; splits: test (191)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of samples) * 100.

Input / output format

Input: A high-resolution image paired with a multiple-choice question (options A–E).

Output: The model must output the selected answer letter (A, B, C, D, or E) enclosed in \boxed{}, optionally preceded by grounding coordinates in JSON format.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        answer = extract_answer_from_box(pred)
        if answer == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Images exceeding 1,003,520 pixels are resized to fit the model's token limit, which can degrade performance on tasks requiring fine-grained details if not handled carefully.
• The evaluation strictly requires the answer letter to be placed within \boxed{} for reliable parsing; deviations cause accuracy calculation failures.
• V* Bench is an out-of-distribution (OOD) benchmark while MME-Realworld is in-distribution (ID); results must be reported separately to avoid conflating generalization with memorization.

Evidence (verbatim from paper)

All datasets employ the multiple-choice question format, and model performance is consistently measured by accuracy on both the in-distribution (MME-Realworld) and out-of-distribution (V* Bench) test sets.

Citation

@misc{huang2025mgpo,
  title={High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning},
  author={Xinyu Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.05920}
}

• arXiv: 2507.05920