# Crbench Eval

> Evaluates whether multimodal large language models can perform genuine visual reasoning on charts by inferring values from axes and scales, rather than relying on OCR or pre-existing annotations. It probes the model's ability to interpret complex visual structures and perform multi-step estimation on both synthetic and real-world charts. Use when the user wants to benchmark on CRBench, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/crbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/crbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/crbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/crbench-eval

---


# crbench-eval

> Do MLLMs Really Understand the Charts? — Xiao Zhang et al. (2025) (arXiv:2509.04457, 2025)

## What this evaluates

Evaluates whether multimodal large language models can perform genuine visual reasoning on charts by inferring values from axes and scales, rather than relying on OCR or pre-existing annotations. It probes the model's ability to interpret complex visual structures and perform multi-step estimation on both synthetic and real-world charts.

## Datasets

- **CRBench** — total ?; splits: Synthetic (-1), Real (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of questions. Calculated as (number of correct predictions / total predictions) * 100. Standard exact-match evaluation is used for chart value estimation tasks.

## Input / output format

**Input**: Chart images (synthetic or real, non-annotated) paired with a natural language question asking for value estimation or chart interpretation.

**Output**: A direct answer or a Chain-of-Thought reasoning trace followed by the final answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize(pred) == normalize(gold):
        correct += 1
accuracy = (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models often bypass visual reasoning by relying on OCR to read axis labels or pre-existing annotations, leading to inflated scores on annotation-heavy benchmarks.
- Applying Chain-of-Thought prompting without underlying visual reasoning capabilities frequently degrades performance, as the model is forced to articulate flawed reasoning steps.

## Evidence (verbatim from paper)

> | Methods | Accuracy | | | 
 | --- | --- | --- | 
 | Synthetic | Real | Overall | The results in Table[1] underscore the significant challenge our CRBench presents to current models. The generally low scores across the board reveal a critical weakness: the widespread inability to reason over non-annotated charts.

## Citation

```bibtex
@misc{zhang2025domllmsreallyunderstandthecharts,
  title={Do MLLMs Really Understand the Charts?},
  author={Xiao Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.04457}
}
```

- arXiv: 2509.04457