Perceptionprocessbench Eval

perceptionprocessbench-eval

Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned — Ong et al. (2025) (arXiv:2509.23250, 2025)

What this evaluates

Evaluates a vision-language process reward model's ability to detect step-level visual grounding and reasoning errors in structured multimodal reasoning traces. The benchmark specifically probes whether the model can distinguish between correct and subtly mutated perception steps that are designed to be challenging for automated error detection.

Datasets

• PerceptionProcessBench — total 2000; splits: test (2000); repo https://github.com/theogbrand/vlprm

Metrics

• step-level correctness (primary) — range: [0, 1]
  • Proportion of correctly classified steps (correct vs. incorrect) out of the total number of evaluated steps. Computed as (number of correct predictions) / (total steps).

Input / output format

Input: Image $I$, question $q$, preceding reasoning steps $s_{<i}$, and candidate step $s_i$.

Output: Binary label: 'correct' or 'incorrect'.

Scoring recipe

correct = 0
total = 0
for step in PerceptionProcessBench:
    pred = model.predict(step.image, step.question, step.context, step.step)
    if pred == step.gold_label:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Synthetic negative examples are deliberately mutated to be subtle, making them significantly harder to detect than obvious errors.
• The underlying training data is highly imbalanced (86% perception errors), so models may bias towards predicting 'correct' if not properly calibrated or balanced during evaluation.

Evidence (verbatim from paper)

assessing their performance on perception error detection requires a dataset with explicit correctness annotations for perception steps. To address this, we synthetically construct PerceptionProcessBench from PuzzleVQA.

Citation

@misc{ong2025vlprm,
  title={Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned},
  author={Ong et al. (2025)},
  year={2025},
  note={arXiv:2509.23250}
}

• arXiv: 2509.23250