Vilbench Eval

vilbench-eval

ViLBench: A Suite for Vision-Language Process Reward Modeling — Tu et al. (2025) (arXiv:2503.20271, 2025)

What this evaluates

Evaluates vision-language models' ability to solve multi-hop visual reasoning tasks by measuring how accurately their final predicted answers match the ground truth. It specifically probes the model's capacity for structured reasoning and answer extraction in complex domains like geometry, science, and visual question answering.

Datasets

• MAVIS-Geometry — total ?; splits: test (-1)
• A-OKVQA — total ?; splits: test (-1)
• GeoQA170K — total ?; splits: test (-1)
• CLEVR-Math — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Accuracy is computed as the proportion of instances where the model's final predicted answer exactly matches the ground truth answer. Answers are extracted using an LLM-based prompt that ignores intermediate reasoning steps and focuses solely on the final letter/answer.

Input / output format

Input: Vision-language prompt or question requiring multi-step reasoning, accompanied by relevant images or visual context.

Output: A generated natural language response containing reasoning steps and a final answer (typically a letter or short string).

Scoring recipe

def compute_accuracy(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        # LLM-based extraction step (simplified)
        final_answer = extract_final_answer(pred) # via GPT-3.5-turbo prompt
        if final_answer == gt:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Relying on regex or simple string matching for answer extraction often fails due to varied model output formats; the paper explicitly uses GPT-3.5-turbo to avoid this.
• Evaluating intermediate reasoning steps instead of just the final answer, which contradicts the protocol's instruction to 'ignore any reasoning or intermediate steps'.
• Not standardizing the ground truth format (e.g., letter vs. full word) before comparison, leading to false negatives.

Evidence (verbatim from paper)

We employ the accuracy between predicted answers and the ground truth as the metric for our ViLBench. To avoid inaccurate extraction of the answer, we follow previous works*[lu2024mathvista, zhang2024mathverse]* to employ GPT-based extraction. In detail, we prompt GPT-3.5-turbo to compare the prediction with the ground truth, the input instruction shows below: {mdframed}[backgroundcolor=pink!15] Given the following:

Generated Answer: model predicted answer

Ground Truth Answer: ground truth answer

Please compare the final answer in the generated response to the ground truth answer. Ignore any reasoning or intermediate steps and focus only on whether the final letter answer in the generated response matches the ground truth.

Output True if the final answer aligns with the ground truth answer; otherwise, output False.

Citation

@misc{tu2025vilbench,
  title={ViLBench: A Suite for Vision-Language Process Reward Modeling},
  author={Tu et al. (2025)},
  year={2025},
  note={arXiv:2503.20271}
}

• arXiv: 2503.20271