V Triune Eval

v-triune-eval

One RL to See Them All: Visual Triple Unified Reinforcement Learning — Yan Ma et al. (2025) (arXiv:2505.18129, 2025)

What this evaluates

Evaluates vision-language models on a unified suite of visual reasoning and perception tasks, measuring generalization across real-world benchmarks, mathematical reasoning, and object detection/grounding capabilities.

Datasets

• MEGA-Bench Core — total ?; splits: test (-1)
• MMMU — total ?; splits: val (-1)
• MathVista — total ?; splits: testmini (-1)
• COCO — total ?; splits: val-2017 (-1)
• OVDEval — total ?; splits: test (-1)
• CountBench — total ?; splits: test (-1)
• OCRBench — total ?; splits: v2 (-1)
• ScreenSpot-Pro — total ?; splits: test (-1)

Metrics

• MEGA-Bench Core weighted average (primary) — range: percent
  • Weighted average score across 440 diverse tasks in the MEGA-Bench Core subset, computed using the official evaluation implementation.
• mAP / mAP@50 — range: percent
  • Mean Average Precision and mAP at IoU threshold 0.5. Computed as the mean of sample-level metrics rather than over the entire dataset due to the absence of confidence scores in model outputs.
• NMS-AP — range: percent
  • Task-wide mean Average Precision after Non-Maximum Suppression, computed as the mean of sample-level metrics.
• Accuracy — range: percent
  • Exact match or rule-based correctness for reasoning and OCR tasks. For MMMU, both rule-based and GPT-4o-based scoring are reported.

Input / output format

Input: Image(s) paired with task-specific text prompts. For detection and grounding tasks, prompts instruct the model to identify instances of a given object and format the output in a specific reasoning-answer format.

Output: Textual reasoning and answers. For perception tasks, bounding box coordinates represented as values relative to the original input image dimensions.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'MEGA-Bench':
        return weighted_average_score(predictions, golds)
    elif task_type in ['COCO', 'OVDEval']:
        sample_scores = [compute_sample_iou(pred, gold) for pred, gold in zip(predictions, golds)]
        return mean(sample_scores)
    elif task_type in ['MMMU', 'MathVista', 'CountBench', 'OCRBench', 'ScreenSpot-Pro']:
        return accuracy(predictions, golds)
    return None

Common pitfalls

• mAP and NMS-AP are computed as the mean of sample-level metrics rather than over the entire dataset due to missing confidence scores.
• MMMU uses both rule-based and GPT-4o scoring, which may introduce variability in results.
• Greedy decoding is used for evaluation, while temperature=1.0 is used during training rollout.

Evidence (verbatim from paper)

We report both IoU and mAP for COCO and OVDEval. Notably, mAP and NMS-mAP are computed as the mean of sample-level metrics, rather than over the entire dataset, due to the absence of confidence scores in model outputs.

Citation

@misc{ma2025one,
  title={One RL to See Them All: Visual Triple Unified Reinforcement Learning},
  author={Yan Ma et al. (2025)},
  year={2025},
  note={arXiv:2505.18129}
}

• arXiv: 2505.18129