qhjqhj00 crbench-eval
Do MLLMs Really Understand the Charts? — Xiao Zhang et al. (2025) (arXiv:2509.04457, 2025)
What this evaluates
Evaluates whether multimodal large language models can perform genuine visual reasoning on charts by inferring values from axes and scales, rather than relying on OCR or pre-existing annotations. It probes the model's ability to interpret complex visual structures and perform multi-step estimation on both synthetic and real-world charts.
Datasets
- CRBench — total ?; splits: Synthetic (-1), Real (-1)
Metrics
Accuracy(primary) — range: percent- Percentage of correctly predicted answers out of the total number of questions. Calculated as (number of correct predictions / total predictions) * 100. Standard exact-match evaluation is used for chart value estimation tasks.
Input / output format
Input: Chart images (synthetic or real, non-annotated) paired with a natural language question asking for value estimation or chart interpretation.
Output: A direct answer or a Chain-of-Thought reasoning trace followed by the final answer.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, gold_answers):
if normalize(pred) == normalize(gold):
correct += 1
accuracy = (correct / len(gold_answers)) * 100
Common pitfalls
- Models often bypass visual reasoning by relying on OCR to read axis labels or pre-existing annotations, leading to inflated scores on annotation-heavy benchmarks.
- Applying Chain-of-Thought prompting without underlying visual reasoning capabilities frequently degrades performance, as the model is forced to articulate flawed reasoning steps.
Evidence (verbatim from paper)
| Methods | Accuracy | | | | --- | --- | --- | | Synthetic | Real | Overall | The results in Table[1] underscore the significant challenge our CRBench presents to current models. The generally low scores across the board reveal a critical weakness: the widespread inability to reason over non-annotated charts.
Citation
@misc{zhang2025domllmsreallyunderstandthecharts,
title={Do MLLMs Really Understand the Charts?},
author={Xiao Zhang et al. (2025)},
year={2025},
note={arXiv:2509.04457}
}
- arXiv: 2509.04457