Chartqa Eval

chartqa-eval

Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding — Ji et al. (2025) (arXiv:2504.09764, 2025)

What this evaluates

Evaluates multimodal models' ability to answer questions about charts by extracting visual and textual information. It probes robustness to missing labels and geometric perturbations, distinguishing between simple pattern matching and true structural reasoning.

Datasets

• ChartQA — total ?; splits: human (-1), augmented (-1)
• Charixv — total ?; splits: test (-1)

Metrics

• Relaxed Accuracy (RA) (primary) — range: percent
  • Measures the accuracy of question answering on the dataset, considering both human-authored and augmented QA pairs. It is the standard metric for chart understanding, typically allowing for minor formatting or unit variations in the predicted answer compared to the ground truth.

Input / output format

Input: Chart image (bar, pie, or line chart) paired with a natural language question.

Output: Natural language answer to the question.

Scoring recipe

def compute_ra(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• High scores on augmented questions often reflect memorization of programmatic patterns rather than genuine chart reasoning.
• Models frequently rely on OCR/text shortcuts instead of visual estimation, causing sharp performance drops under label removal or perturbation.
• Aggregating human and augmented sets can mask poor performance on complex, real-world chart questions.

Evidence (verbatim from paper)

In terms of evaluation metric, we use Relaxed Accuracy (RA), which measures the accuracy of question answering on the ChartQA dataset, considering both human-authored and augmented QA pairs. This is the standard evaluation metric adopted by previous chart understanding works, enabling direct and fair comparison across different approaches.

Citation

@misc{ji2025socraticchart,
  title={Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.09764}
}

• arXiv: 2504.09764