Charxiv Eval

charxiv-eval

CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs — Wang et al. (2024) (arXiv:2406.18521, 2024)

What this evaluates

Evaluates multimodal large language models' ability to understand real-world charts through descriptive and reasoning tasks. It probes capabilities like information extraction, pattern recognition, counting, compositional reasoning, and robustness to chart complexity (e.g., multiple subplots) and unanswerable queries.

Datasets

• CharXiv — total 2323; splits: validation (-1), test (-1); repo https://github.com/princeton-nlp/CharXiv

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions in a category. Evaluated via exact match or automated grading consistent with human evaluation protocols.

Input / output format

Input: A chart image (minimum resolution 448×448) paired with a natural language question (either descriptive or reasoning).

Output: A natural language text string containing the model's predicted answer.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, golds):
    if normalize(pred) == normalize(gold):
        correct += 1
accuracy = (correct / len(golds)) * 100

Common pitfalls

• Models perform near-randomly on compositional tasks like counting axis ticks, despite humans succeeding easily.
• Performance degrades sharply (30-50% drop) for open-source models when charts contain 6 or more subplots.
• Weak models frequently fail to identify unanswerable questions, often hallucinating answers instead.

Evidence (verbatim from paper)

Although counting is easy for humans, this particular task causes 20 out of 24 models to achieve an accuracy below 10% (our random baseline achieves 5.35%).

Citation

@misc{wang2024charxiv,
  title={CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.18521}
}

• arXiv: 2406.18521