Virc Multimodal Math Eval

virc-multimodal-math-eval

ViRC: Enhancing Visual Interleaved Mathematical CoT with Reason Chunking — Lihong Wang et al. (2025) (arXiv:2512.14654, 2025)

What this evaluates

Evaluates multimodal mathematical reasoning and high-resolution visual perception capabilities of vision-language models. It probes the model's ability to decompose complex problems into structured reasoning chunks, interleave visual tool calls, and produce accurate final answers across geometric, mathematical, and fine-grained visual benchmarks.

Datasets

• GeoQA — total ?; splits: test (-1)
• MathVista-Math — total ?; splits: test (-1)
• MMStar-Math — total ?; splits: test (-1)
• VisualProbe — total ?; splits: test (-1)
• V* — total ?; splits: test (-1)
• HR-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match accuracy calculated as the fraction of correctly answered instances out of the total number of instances, reported as a percentage or decimal. Final answers are extracted from the model's output and compared against ground-truth labels.

Input / output format

Input: Multimodal input consisting of one or more images (up to 16K resolution) paired with a text-based mathematical or visual perception question.

Output: A structured reasoning chain interleaved with visual tool calls (e.g., crop, scale, display) followed by a final answer string.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        pred_answer = extract_final_answer(pred)
        if normalize_string(pred_answer) == normalize_string(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• High-resolution images (2K-16K) require careful tokenization and memory management; naive resizing or cropping can destroy fine-grained details needed for correct answers.
• Extracting sub-datasets like MathVista-Math and MMStar-Math requires strict adherence to the original paper's split definitions to avoid evaluation mismatch or data leakage.
• Parsing interleaved tool calls and reasoning steps for final answer extraction can be error-prone if the model's output format varies across benchmarks.

Evidence (verbatim from paper)

The proposed VIRC is mainly evaluated on mathematical benchmarks. GeoQA [3] is a geometric problem benchmark with annotated solution programs evaluated on the Geo170K test set consistent with R1-V [5] and Hint-GRPO [13]. MathVista-Math and MMStar-Math are extracted from MathVista [21] and MMStar [4], respectively, as mathematical capability dimensions, consistent with MINT-CoT [6]. While initially designed for the mathematical domain, our method exhibits strong generalization capabilities for vision tasks requiring fine-grained perception. Therefore, we assess V1RC on a suite of high-resolution benchmarks including $V^{*}$ [43], HR-Bench [39], and VisualProbe [17]. These benchmarks are characterized by image resolutions ranging from 2K to 16K. As shown in Tab. 1, our method, VIRC with Reason Chunking, significantly improves the performance of baseline models on multimodal mathematical benchmarks. It achieves an average accuracy gain of 18.52% over Qwen2.5-VL-3B-Instruct and 18.80% over Qwen2.5-VL-7B-Instruct.

Citation

@misc{wang2025virc,
  title={ViRC: Enhancing Visual Interleaved Mathematical CoT with Reason Chunking},
  author={Lihong Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.14654}
}

• arXiv: 2512.14654