qhjqhj00 veroeval-eval
Vero: An Open RL Recipe for General Visual Reasoning — Sarch et al. (2026) (arXiv:2604.04917, 2026)
What this evaluates
Evaluates general visual reasoning capabilities across a diverse set of 30 benchmarks spanning six task categories, including chart/OCR, STEM, spatial/action, knowledge/recognition, grounding, and captioning/instruction following.
Datasets
- VeroEval — total ?; splits: test (-1); repo https://github.com/zlab-princeton/vero
Metrics
overall averages(primary) — range: percent- Mean of the official scores reported by each of the 30 constituent benchmarks. Scores are aggregated at the task-category level and overall.
Input / output format
Input: Multimodal prompts (images + text) per benchmark, evaluated with one sampled decode per example.
Output: Model-generated text responses (answers, reasoning traces, or captions) depending on the benchmark.
Scoring recipe
scores = []
for bench in veroeval_benchmarks:
preds = model.generate(prompts[bench], max_tokens=..., temperature=...)
score = lmms_eval.evaluate(bench, preds, gold) # Official protocol per benchmark
scores.append(score)
overall_averages = sum(scores) / len(scores)
Common pitfalls
- Using different decoding parameters than specified per model family (qwen25vl vs qwen3vl).
- Not following each benchmark's official evaluation protocol, which varies by task type.
- LLM/VLM judge prompts and sampling parameters (Temp=0.7, TopP=0.8) can introduce variance if not fixed.
Evidence (verbatim from paper)
Our best models Vero-Qwen3T-8B and Vero-Qwen3I-8B achieve the highest overall averages (65.9 and 66.0, respectively) among all 8B-parameter VLMs evaluated, outperforming baselines across the six task categories. We evaluate all models using the lmms-eval framework, following the official evaluation protocols specified by each benchmark’s authors.
Citation
@misc{sarch2026vero,
title={Vero: An Open RL Recipe for General Visual Reasoning},
author={Sarch et al. (2026)},
year={2026},
note={arXiv:2604.04917}
}
- arXiv: 2604.04917