Chartverse Eval

chartverse-eval

ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch — Zheng Liu et al. (arXiv:2601.13606, 2026)

What this evaluates

Evaluates visual language models on complex chart understanding and reasoning tasks, including data extraction, comparison, and trend analysis across diverse chart types and domains.

Datasets

• ChartQA-Pro — total ?; splits: test (-1)
• CharXiv — total ?; splits: test (-1)
• ChartMuseum — total ?; splits: test (-1)
• ChartX — total ?; splits: test (-1)
• ChartBench — total ?; splits: test (-1)
• EvoChart — total ?; splits: test (-1)

Metrics

• average score (primary) — range: percent
  • Arithmetic mean of the per-benchmark accuracy or score values across all evaluated chart reasoning datasets.

Input / output format

Input: Chart image (PNG/JPG) paired with a natural language question requiring reasoning or data extraction.

Output: Natural language text containing the final answer, often preceded by chain-of-thought reasoning.

Scoring recipe

scores = []
for dataset in [ChartQA-Pro, CharXiv, ChartMuseum, ChartX, ChartBench, EvoChart]:
    preds = model.generate(image, question)
    acc = compute_accuracy(preds, gold_answers)
    scores.append(acc)
average_score = sum(scores) / len(scores)
return average_score

Common pitfalls

• The reported 'average score' aggregates heterogeneous metrics from different benchmarks without specifying normalization or weighting schemes.
• The paper does not disclose exact evaluation prompts, temperature settings, or decoding parameters used during inference.

Evidence (verbatim from paper)

We evaluate performance across 6 benchmarks necessitating complex chart understanding and reasoning: ChartQA-Pro [[26]], CharXiv [[35]], ChartMuseum [[33]], ChartX [[36]], ChartBench [[40]], and EvoChart [[9]]. ... ChartVerse-2B achieves an average score of 54.3, exceeding all chart-specific baselines, including ECD-7B, START-7B, and Chart-R1-7B.

Citation

@misc{liu2026chartverse,
  title={ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch},
  author={Zheng Liu et al.},
  year={2026},
  note={arXiv:2601.13606}
}

• arXiv: 2601.13606