# Chainv Eval

> Evaluates the accuracy and inference efficiency of training-free multimodal reasoning methods across diverse vision-language benchmarks. It probes how well atomic visual hint injection reduces redundant reasoning steps while maintaining or improving task performance on math, logic, science, and general visual understanding tasks. Use when the user wants to benchmark on MathVista mini, MathVision, WeMath, MMMU Pro vis, LogicVista, OlympiadBench, VStar, CVBench, ConBench, ChartVQA, SEED-Bench, ScreenSpot, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/chainv-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/chainv-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/chainv-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/chainv-eval

---


# chainv-eval

> ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better — Yuan Zhang et al. (2025) (arXiv:2511.17106, 2025)

## What this evaluates

Evaluates the accuracy and inference efficiency of training-free multimodal reasoning methods across diverse vision-language benchmarks. It probes how well atomic visual hint injection reduces redundant reasoning steps while maintaining or improving task performance on math, logic, science, and general visual understanding tasks.

## Datasets

- **MathVista mini** — total ?; splits: test (-1)
- **MathVision** — total ?; splits: test (-1)
- **WeMath** — total ?; splits: test (-1)
- **MMMU Pro vis** — total ?; splits: test (-1)
- **LogicVista** — total ?; splits: test (-1)
- **OlympiadBench** — total ?; splits: test (-1)
- **VStar** — total ?; splits: test (-1)
- **CVBench** — total ?; splits: test (-1)
- **ConBench** — total ?; splits: test (-1)
- **ChartVQA** — total ?; splits: test (-1)
- **SEED-Bench** — total ?; splits: test (-1)
- **ScreenSpot** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of samples in the benchmark split.
- `Inference Time (s)` — range: seconds
  - Average wall-clock time per sample, explicitly including tokenizer, prefill, decoding loop, and post-processing.
- `accuracy per second (acc/s)` — range: other
  - Accuracy divided by average inference time per sample, used to measure method efficiency.

## Input / output format

**Input**: Image(s) paired with a text prompt/question requiring multimodal reasoning, visual understanding, OCR, or GUI navigation.

**Output**: Text response generated by the model, typically a direct answer or step-by-step reasoning trace.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, inference_times):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    accuracy = (correct / len(gold_answers)) * 100
    avg_time = sum(inference_times) / len(inference_times)
    acc_per_sec = accuracy / avg_time if avg_time > 0 else 0
    return {'accuracy': accuracy, 'inference_time_s': avg_time, 'acc_per_sec': acc_per_sec}
```

## Common pitfalls

- Inference time must include the full pipeline (tokenizer, prefill, decoding, post-processing), not just the autoregressive decoding loop.
- Accuracy is reported on specific subsets (e.g., MathVista mini, MMMU Pro vis), not the full benchmark suites.
- Latency comparisons require fixed random seeds, batch sizes, and GPU memory utilization to avoid hardware-induced fluctuations.

## Evidence (verbatim from paper)

> On average, ChainV brings $+2.1\sim 3.5$ absolute improvement in accuracy over the baseline models while outperforming other comparative methods.

## Citation

```bibtex
@misc{zhang2025chainv,
  title={ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better},
  author={Yuan Zhang et al. (2025)},
  year={2025},
  note={arXiv:2511.17106}
}
```

- arXiv: 2511.17106