deepvision-103k-eval
DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning — Sun et al. (2026) (arXiv:2602.16742, 2026)
What this evaluates
Evaluates the multimodal mathematical reasoning and general multimodal reasoning capabilities of vision-language models. It probes visual perception, step-by-step logical deduction, and cross-domain generalization on K12-level math and broader visual tasks.
Datasets
- Multimodal Math & General Reasoning Benchmarks (WeMath, MathVerse_vision, MathVision, LogicVista, MMMU_VAL, MMMU_Pro_full, M^3CoT) — total ?; splits: test (-1)
Metrics
accuracy (primary) — range: percent
- Percentage of correctly answered questions. Calculated as the number of predictions matching the ground truth answer divided by the total number of evaluation instances, multiplied by 100.
Input / output format
Input: Multimodal inputs consisting of mathematical problem images paired with text prompts/questions.
Output: Textual responses containing the final answer and/or step-by-step reasoning.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, gold_labels):
if normalize_answer(pred) == normalize_answer(gold):
correct += 1
accuracy = (correct / len(predictions)) * 100
Common pitfalls
- Confusing the training reward signal (rule-based +1/0 for answer correctness) with the evaluation metric.
- Failing to use the official decoding parameters and 32K max token limit, which significantly impacts performance on these benchmarks.
- Not accounting for varying answer formats across benchmarks (e.g., multiple-choice vs. open-ended) before applying exact-match scoring.
Evidence (verbatim from paper)
We evaluated our models on the following benchmarks: (1) Multimodal Math: WeMath, MathVerse_vision, MathVision, and LogicVista. (2) General Multimodal: MMMU_VAL, MMMU_Pro_full and M^3CoT. For inference parameters, we set the maximum token length at 32K for all evaluation. Decoding parameters follow the official recommendations. ... Qwen3-VL-8B-DeepVision reaches sota results on WeMath (85.11%), MiMo-VL-7B-DeepVision reaches sota results on LogicVista (65.62%.
Citation
@misc{sun2026deepvision103k,
title={DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning},
author={Sun et al. (2026)},
year={2026},
note={arXiv:2602.16742}
}
1---2name: deepvision-103k-eval3description: Evaluates the multimodal mathematical reasoning and general multimodal reasoning capabilities of vision-language models. It probes visual perception, step-by-step logical deduction, and cross-domain generalization on K12-level math and broader visual tasks. Use when the user wants to benchmark on Multimodal Math & General Reasoning Benchmarks (WeMath, MathVerse_vision, MathVision, LogicVista, MMMU_VAL, MMMU_Pro_full, M^3CoT), or asks about evaluating this task. Reports accuracy.4---56# deepvision-103k-eval78> DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning — Sun et al. (2026) (arXiv:2602.16742, 2026)910## What this evaluates1112Evaluates the multimodal mathematical reasoning and general multimodal reasoning capabilities of vision-language models. It probes visual perception, step-by-step logical deduction, and cross-domain generalization on K12-level math and broader visual tasks.1314## Datasets1516- **Multimodal Math & General Reasoning Benchmarks (WeMath, MathVerse_vision, MathVision, LogicVista, MMMU_VAL, MMMU_Pro_full, M^3CoT)** — total ?; splits: test (-1)1718## Metrics1920- `accuracy` **(primary)** — range: percent21 - Percentage of correctly answered questions. Calculated as the number of predictions matching the ground truth answer divided by the total number of evaluation instances, multiplied by 100.2223## Input / output format2425**Input**: Multimodal inputs consisting of mathematical problem images paired with text prompts/questions.2627**Output**: Textual responses containing the final answer and/or step-by-step reasoning.2829## Scoring recipe3031```python32correct = 033for pred, gold in zip(predictions, gold_labels):34 if normalize_answer(pred) == normalize_answer(gold):35 correct += 136accuracy = (correct / len(predictions)) * 10037```3839## Common pitfalls4041- Confusing the training reward signal (rule-based +1/0 for answer correctness) with the evaluation metric.42- Failing to use the official decoding parameters and 32K max token limit, which significantly impacts performance on these benchmarks.43- Not accounting for varying answer formats across benchmarks (e.g., multiple-choice vs. open-ended) before applying exact-match scoring.4445## Evidence (verbatim from paper)4647> We evaluated our models on the following benchmarks: (1) Multimodal Math: WeMath, MathVerse_vision, MathVision, and LogicVista. (2) General Multimodal: MMMU_VAL, MMMU_Pro_full and M^3CoT. For inference parameters, we set the maximum token length at 32K for all evaluation. Decoding parameters follow the official recommendations. ... Qwen3-VL-8B-DeepVision reaches sota results on WeMath (85.11%), MiMo-VL-7B-DeepVision reaches sota results on LogicVista (65.62%.4849## Citation5051```bibtex52@misc{sun2026deepvision103k,53 title={DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning},54 author={Sun et al. (2026)},55 year={2026},56 note={arXiv:2602.16742}57}58```5960- arXiv: 2602.16742
qhjqhj00