# Test Time Scaling Vlm Eval

> Evaluates the impact of test-time scaling (TTS) inference strategies on Vision-Language Models across multimodal reasoning and perception tasks. It measures how techniques like Chain-of-Thought, Best-of-N, Self-Consistency, and Self-Refinement improve or degrade performance on open-source versus closed-source models. Use when the user wants to benchmark on MathVista, MMMU, MMBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/test-time-scaling-vlm-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/test-time-scaling-vlm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/test-time-scaling-vlm-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/test-time-scaling-vlm-eval

---


# test-time-scaling-vlm-eval

> Limits and Gains of Test-Time Scaling in Vision-Language Reasoning — Ahmadpour et al. (2025) (arXiv:2512.11109, 2025)

## What this evaluates

Evaluates the impact of test-time scaling (TTS) inference strategies on Vision-Language Models across multimodal reasoning and perception tasks. It measures how techniques like Chain-of-Thought, Best-of-N, Self-Consistency, and Self-Refinement improve or degrade performance on open-source versus closed-source models.

## Datasets

- **MathVista** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the dataset.

## Input / output format

**Input**: Multimodal instances consisting of an image and a text question/prompt. For TTS methods, the same instance is processed using different inference strategies (Zero-Shot, CoT, Best-of-N, Self-Consistency, Beam Search, Self-Refinement).

**Output**: Text-based answer (typically multiple-choice or short-form) generated by the VLM. For verification-based methods, an external verifier model may also produce a ranking or selection.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Assuming test-time scaling methods like Self-Refinement or CoT universally improve performance; they often degrade open-source models or perception-heavy benchmarks.
- Attempting to use internal confidence scores for closed-source models (e.g., Gemini, GPT-4o), which do not expose token-level probabilities.
- Expecting stochastic decoding to help on deterministic visual tasks where models are already near saturation.

## Evidence (verbatim from paper)

> The evaluation metrics for all datasets are accuracy scores, representing the percentage of correctly answered questions.

## Citation

```bibtex
@misc{ahmadpour2025testtimescaling,
  title={Limits and Gains of Test-Time Scaling in Vision-Language Reasoning},
  author={Ahmadpour et al. (2025)},
  year={2025},
  note={arXiv:2512.11109}
}
```

- arXiv: 2512.11109