Chainv Eval

chainv-eval

ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better — Yuan Zhang et al. (2025) (arXiv:2511.17106, 2025)

What this evaluates

Evaluates the accuracy and inference efficiency of training-free multimodal reasoning methods across diverse vision-language benchmarks. It probes how well atomic visual hint injection reduces redundant reasoning steps while maintaining or improving task performance on math, logic, science, and general visual understanding tasks.

Datasets

• MathVista mini — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)
• MMMU Pro vis — total ?; splits: test (-1)
• LogicVista — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)
• VStar — total ?; splits: test (-1)
• CVBench — total ?; splits: test (-1)
• ConBench — total ?; splits: test (-1)
• ChartVQA — total ?; splits: test (-1)
• SEED-Bench — total ?; splits: test (-1)
• ScreenSpot — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of samples in the benchmark split.
• Inference Time (s) — range: seconds
  • Average wall-clock time per sample, explicitly including tokenizer, prefill, decoding loop, and post-processing.
• accuracy per second (acc/s) — range: other
  • Accuracy divided by average inference time per sample, used to measure method efficiency.

Input / output format

Input: Image(s) paired with a text prompt/question requiring multimodal reasoning, visual understanding, OCR, or GUI navigation.

Output: Text response generated by the model, typically a direct answer or step-by-step reasoning trace.

Scoring recipe

def compute_metrics(predictions, gold_answers, inference_times):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    accuracy = (correct / len(gold_answers)) * 100
    avg_time = sum(inference_times) / len(inference_times)
    acc_per_sec = accuracy / avg_time if avg_time > 0 else 0
    return {'accuracy': accuracy, 'inference_time_s': avg_time, 'acc_per_sec': acc_per_sec}

Common pitfalls

• Inference time must include the full pipeline (tokenizer, prefill, decoding, post-processing), not just the autoregressive decoding loop.
• Accuracy is reported on specific subsets (e.g., MathVista mini, MMMU Pro vis), not the full benchmark suites.
• Latency comparisons require fixed random seeds, batch sizes, and GPU memory utilization to avoid hardware-induced fluctuations.

Evidence (verbatim from paper)

On average, ChainV brings $+2.1\sim 3.5$ absolute improvement in accuracy over the baseline models while outperforming other comparative methods.

Citation

@misc{zhang2025chainv,
  title={ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better},
  author={Yuan Zhang et al. (2025)},
  year={2025},
  note={arXiv:2511.17106}
}

• arXiv: 2511.17106