Pyvision Rl Eval

pyvision-rl-eval

PyVision-RL: Forging Open Agentic Vision Models via RL — Zhao et al. (2026) (arXiv:2602.20739, 2026)

What this evaluates

Evaluates open-weight multimodal agentic models on visual search, multimodal mathematical reasoning, multi-turn tool use, and video spatial reasoning. It probes the model's ability to dynamically construct context, invoke tools, and perform long-horizon reasoning with high visual token efficiency.

Datasets

• V* — total 191; splits: test (191)
• HRBench-4K — total ?; splits: test (-1)
• HRBench-8K — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)
• DynaMath — total ?; splits: test (-1)
• TIR-Bench — total ?; splits: test (-1)
• VSI-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or successfully completed tasks. Calculated as (number of correct predictions / total number of samples) * 100.
• avg@32 — range: percent
  • Average performance metric reported specifically for the V* benchmark due to its small dataset size (191 samples). Represents the average score over 32 samples or queries.

Input / output format

Input: Image or video input paired with a text prompt/question. For video evaluation, frames are dynamically sampled via Python code based on the model's reasoning rather than using fixed FPS or uniform sampling.

Output: Text response containing the final answer, tool calls, or Python code for frame retrieval. Generation uses temperature 0.01 for V* and temperature 0.5 with top-k 20 for other benchmarks, with a maximum turn budget of 30 and maximum context length of 32K tokens.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

def compute_avg32(predictions, gold, k=32):
    # For V*, average performance over k samples
    return compute_accuracy(predictions[:k], gold[:k])

Common pitfalls

• V* uses avg@32 instead of standard accuracy due to its small size (191 samples).
• Video evaluation uses dynamic, Python-based on-demand frame sampling, not fixed FPS or uniform sampling.
• Generation hyperparameters differ by benchmark: V* uses temperature 0.01, while others use temperature 0.5 with top-k 20.

Evidence (verbatim from paper)

During evaluation, PyVision-Image uses a temperature of 0.01 for V* and 0.5 with top-k 20 for the other benchmarks, whereas PyVision-Video uses a temperature of 0.01. Given the long-horizon reasoning capabilities induced by RL tuning, we set the maximum turn budget to 30 and the maximum context length to 32K tokens. We evaluate our models on the following benchmarks: Visual Search. To assess the model's agentic visual perception capabilities, we evaluate our model on V* (Wu & Xie, 2024), HRBench-4K (Wang et al., 2025b), and HRBench8K (Wang et al., 2025b). Since V contains only 191 samples, we report results using the avg@32 metric.

Citation

@misc{zhao2026pyvisionrl,
  title={PyVision-RL: Forging Open Agentic Vision Models via RL},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2602.20739}
}

• arXiv: 2602.20739