Test Time Scaling Vlm Eval

test-time-scaling-vlm-eval

Limits and Gains of Test-Time Scaling in Vision-Language Reasoning — Ahmadpour et al. (2025) (arXiv:2512.11109, 2025)

What this evaluates

Evaluates the impact of test-time scaling (TTS) inference strategies on Vision-Language Models across multimodal reasoning and perception tasks. It measures how techniques like Chain-of-Thought, Best-of-N, Self-Consistency, and Self-Refinement improve or degrade performance on open-source versus closed-source models.

Datasets

• MathVista — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions in the dataset.

Input / output format

Input: Multimodal instances consisting of an image and a text question/prompt. For TTS methods, the same instance is processed using different inference strategies (Zero-Shot, CoT, Best-of-N, Self-Consistency, Beam Search, Self-Refinement).

Output: Text-based answer (typically multiple-choice or short-form) generated by the VLM. For verification-based methods, an external verifier model may also produce a ranking or selection.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Assuming test-time scaling methods like Self-Refinement or CoT universally improve performance; they often degrade open-source models or perception-heavy benchmarks.
• Attempting to use internal confidence scores for closed-source models (e.g., Gemini, GPT-4o), which do not expose token-level probabilities.
• Expecting stochastic decoding to help on deterministic visual tasks where models are already near saturation.

Evidence (verbatim from paper)

The evaluation metrics for all datasets are accuracy scores, representing the percentage of correctly answered questions.

Citation

@misc{ahmadpour2025testtimescaling,
  title={Limits and Gains of Test-Time Scaling in Vision-Language Reasoning},
  author={Ahmadpour et al. (2025)},
  year={2025},
  note={arXiv:2512.11109}
}

• arXiv: 2512.11109