Chartgalaxy Eval

chartgalaxy-eval

ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation — Li et al. (2025) (arXiv:2505.18668, 2025)

What this evaluates

This benchmark evaluates multimodal large language models' ability to understand and reason about infographic charts. It probes capabilities in text-based data reasoning, visual-element association, and visual style analysis through structured question-answering tasks.

Datasets

• ChartGalaxy — total 70248; splits: train (70248), test (2176); repo https://github.com/ChartGalaxy/ChartGalaxy

Metrics

• relaxed accuracy (5% margin) (primary) — range: percent
  • Accuracy calculated by comparing model predictions to ground truth, allowing a 5% relative tolerance margin for numerical answers. Textual answers are evaluated using ANLS, and multiple-choice questions use exact matching.

Input / output format

Input: An infographic chart image paired with a natural language question (covering data identification, comparison, extraction, fact-checking, visual-element association, or visual understanding).

Output: A natural language answer string, a selected multiple-choice option, or a numerical value.

Scoring recipe

def compute_vqa_accuracy(predictions, golds, question_types):
    correct = 0
    for pred, gold, qtype in zip(predictions, golds, question_types):
        if qtype == 'multiple_choice':
            if pred.strip().lower() == gold.strip().lower(): correct += 1
        elif qtype == 'text':
            if compute_anls(pred, gold) >= 0.5: correct += 1
        elif qtype == 'numerical':
            if abs(float(pred) - float(gold)) / float(gold) <= 0.05: correct += 1
    return correct / len(predictions) * 100

Common pitfalls

• Numerical answers require a strict 5% relative margin tolerance; absolute differences can lead to false negatives.
• ANLS for textual answers uses a standard 0.5 threshold, but variations in tokenization or case sensitivity can affect scores if not normalized.
• The independent evaluation set only contains synthetic charts with bounding-box annotations, so results may not generalize to real-world infographic charts without visual elements.

Evidence (verbatim from paper)

For the evaluation metrics, we follow previous work on chart question answering*(Masry et al., [2025])*, using relaxed accuracy with a 5% margin for numerical answers, ANLS for textual answers, and exact matching for multiple-choice questions.

Citation

@misc{li2025chartgalaxy,
  title={ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.18668}
}

• arXiv: 2505.18668