Chartmuseum Eval

chartmuseum-eval

ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models — Tang et al. (2025) (arXiv:2505.13444, 2025)

What this evaluates

This benchmark evaluates the visual reasoning capabilities of Large Vision-Language Models (LVLMs) on real-world charts. It specifically probes the model's ability to perform visual extraction, object picking, visual comparisons, and trajectory tracking, while distinguishing these from purely textual inference tasks.

Datasets

• CHARTMUSEUM — total 1162; splits: Visual (510), Synthesis (133), Visual/Text (234), Text (123), Overall (1000)

Metrics

• accuracy (primary) — range: percent
  • Accuracy is computed as the percentage of questions answered correctly. Answers are evaluated using an LLM-as-a-Judge approach to handle paraphrasing, as all questions are designed to have unique, objective answers.

Input / output format

Input: A chart image paired with a natural language question requiring visual or textual reasoning.

Output: A unique, objective answer string (e.g., a specific value, category name, or short phrase).

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # LLM-as-a-Judge evaluates semantic match using Appendix Fig 27 prompt
        if llm_judge_match(pred, gold):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models heavily over-rely on OCR/textual extraction rather than actual visual reasoning, leading to artificially high scores on 'Text' splits but poor performance on 'Visual' splits.
• Applying chain-of-thought or extended reasoning prompts yields minimal gains for chart understanding compared to math/code tasks, contrary to recent LLM trends.
• LLM-as-a-Judge evaluation requires a carefully crafted prompt to avoid bias; naive exact-match fails due to paraphrasing.

Evidence (verbatim from paper)

Table 3: Accuracy performance comparison of models on the test set of the CHARTMUSEUM benchmark. Humans achieve an overall accuracy of 93.0% on a randomly sampled subset consisting 100 examples.

Citation

@misc{tang2025chartmuseum,
  title={ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2505.13444}
}

• arXiv: 2505.13444