Openvlthinkerv2 Eval

openvlthinkerv2-eval

OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks — Wenbo Hu et al. (arXiv:2604.08539, 2026)

What this evaluates

Evaluates a multimodal reasoning model's capability across diverse visual tasks, including general and mathematical VQA, document understanding, spatial reasoning, and visual grounding. The protocol tests the model's ability to balance fine-grained perception with multi-step reasoning under a unified RL training framework.

Datasets

• MMMU — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• EmbSpatial — total ?; splits: test (-1)
• RefSpatial — total ?; splits: test (-1)
• RoboSpatial — total ?; splits: test (-1)
• RefCOCO — total ?; splits: val (-1)
• RefCOCO+ — total ?; splits: val (-1)
• RefCOCOg — total ?; splits: val (-1)

Metrics

• score (primary) — range: percent
  • Standard accuracy or composite benchmark score reported as a percentage or raw value depending on the specific benchmark. Calculated as the proportion of correctly predicted answers or bounding boxes matching ground truth within a specified threshold.

Input / output format

Input: Multimodal inputs consisting of an image and a text prompt/question.

Output: Text response containing the answer, reasoning chain, or bounding box coordinates.

Scoring recipe

def calculate_score(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if is_correct(pred, gold_item):
            correct += 1
    return (correct / len(gold)) * 100

def is_correct(pred, gold):
    if gold_type == 'bbox':
        return compute_iou(pred, gold) >= 0.5
    return pred.strip().lower() == gold.strip().lower()

Common pitfalls

• Models must be evaluated using identical generation hyperparameters to baselines to ensure fair comparison.
• Reward shaping and data filtering during RL training can cause overfitting to specific task distributions, inflating scores on in-domain benchmarks while degrading out-of-distribution generalization.
• Composite benchmarks like OCRBench aggregate multiple sub-tasks, making it difficult to isolate specific capability improvements.

Evidence (verbatim from paper)

Table 3: Evaluation results for Grounding task. We report scores on val splits. Our model consistently outperforms previous baselines.

Citation

@misc{hu2026openvlthinkerv2,
  title={OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks},
  author={Wenbo Hu et al.},
  year={2026},
  note={arXiv:2604.08539}
}

• arXiv: 2604.08539