Gsm8k V Eval

gsm8k-v-eval

GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts — Fan Yuan et al. (2025) (arXiv:2509.25160, 2025)

What this evaluates

This benchmark evaluates vision-language models' ability to perform multi-step mathematical reasoning using purely visual, comic-style narratives instead of text. It specifically probes challenges in inter-image semantic understanding, object grounding, and extracting numerical relationships from multi-panel visual contexts.

Datasets

• GSM8K-V — total ?; splits: test (-1); repo https://github.com/ZJU-REAL/GSM8K-V

Metrics

• accuracy (primary) — range: percent
  • Percentage of problems where the model's final predicted numeric answer exactly matches the ground truth answer. Calculated as (number of correct predictions / total number of problems) * 100.

Input / output format

Input: Multi-image comic-style visual narratives depicting grade school math word problems. Models receive a sequence of images representing the problem context and are prompted to reason step by step.

Output: Only the final numeric answer in the required format (integer, decimal, or fraction), without additional reasoning text.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Normalize whitespace and numeric formats
        pred_val = normalize_numeric(pred.strip())
        gold_val = normalize_numeric(gold.strip())
        if pred_val == gold_val:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models often fail to maintain semantic consistency across multiple comic panels, leading to incorrect intermediate values or misaligned object grounding.
• Strict output formatting is required; models that output reasoning text alongside the answer may cause parsing failures if not explicitly stripped before scoring.
• Visual ambiguity in comic-style representations can cause models to misidentify quantities or relationships, resulting in systematic category-specific drops (e.g., 'Other' category).

Evidence (verbatim from paper)

For GSM8K-V, models are instructed to reason step by step over the images and report only the final numeric answer in the required format (integer, decimal, or fraction). The best-performing model, Gemini-2.5-Pro, achieved only 46.93% accuracy, while other models—including flagship ones such as GPT-5 and Llama-4-17B-128E—reached merely around 30%.

Citation

@misc{yuan2025gsm8kv,
  title={GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts},
  author={Fan Yuan et al. (2025)},
  year={2025},
  note={arXiv:2509.25160}
}

• arXiv: 2509.25160