Chart Reasoning Eval

chart-reasoning-eval

CharTool: Tool-Integrated Visual Reasoning for Chart Understanding — Situo Zhang et al. (arXiv:2604.02794, 2026)

What this evaluates

Evaluates a model's ability to understand complex chart visualizations and perform fine-grained visual grounding and numerical reasoning. It probes both in-domain chart comprehension across real-world and synthetic datasets, and out-of-domain generalization to visual mathematical reasoning tasks.

Datasets

• CharXiv — total ?; splits: test (-1)
• ChartQAPro — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• ChartBench — total ?; splits: test (-1)
• ChartX — total ?; splits: test (-1)
• ReachQA — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match or normalized string matching accuracy calculated per sub-task (reasoning, description, recognition, etc.) and averaged across benchmarks. Formula: (Number of correct predictions / Total number of predictions) * 100.

Input / output format

Input: A chart image (bar, line, pie, scatter, or multi-subplot scientific figures) paired with a natural language question or query.

Output: Natural language text containing step-by-step reasoning and a final answer.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Benchmarks report multiple sub-task scores (e.g., reasoning, description, recognition); averaging them without explicit weighting can obscure performance on specific capabilities.
• Out-of-domain benchmarks (MathVista, WeMath, MathVerse) test general visual/math reasoning rather than chart-specific skills, so direct score comparison with chart benchmarks is misleading.
• The paper uses GPT-5.2 for dataset quality evaluation, which is separate from the model evaluation protocol and should not be conflated with the main accuracy metrics.

Evidence (verbatim from paper)

We conduct a comprehensive evaluation of our models across a diverse set of standard chart benchmarks, categorized into: (1) Real-world Benchmarks, comprising charts extracted from real-world sources, including CharXiv, ChartQAPro, and ChartQA; and (2) Synthetic Benchmarks, containing synthesized images or queries: ChartBench, ChartX, and ReachQA. Furthermore, to assess the out-of-domain generalization capabilities of our approach, we include three visual math reasoning benchmarks: MathVista, WeMath, and MathVerse.

Citation

@misc{zhang2026chartool,
  title={CharTool: Tool-Integrated Visual Reasoning for Chart Understanding},
  author={Situo Zhang et al.},
  year={2026},
  note={arXiv:2604.02794}
}

• arXiv: 2604.02794