Refchartqa Eval

refchartqa-eval

RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning — Vogel et al. (2025) (arXiv:2503.23131, 2025)

What this evaluates

This benchmark evaluates a model's ability to answer questions about chart images while simultaneously localizing the visual evidence (via bounding boxes) that supports the answer. It probes spatial-text alignment, arithmetic and logical reasoning over charts, and hallucination reduction through explicit grounding.

Datasets

• RefChartQA — total 73702; splits: train (55789), val (6223), test (11690); repo https://github.com/moured/RefChartQA

Metrics

• answer accuracy (primary) — range: [0, 1]
  • Exact-match accuracy comparing the model's predicted answer string to the ground truth answer. The paper emphasizes exact correctness to reduce hallucination during grounding validation.
• grounding accuracy — range: [0, 1]
  • Accuracy of the predicted bounding box annotation relative to the ground truth grounding element. Validated via manual review showing 93.45% inter-annotator agreement.

Input / output format

Input: A chart image paired with a natural language question (instruction) requiring data retrieval, arithmetic computation, or logical comparison.

Output: A predicted answer string and a bounding box annotation identifying the chart element(s) used to derive the answer.

Scoring recipe

def score(predictions, golds):
    correct_answers = sum(1 for p, g in zip(predictions['answers'], golds['answers']) if p.strip() == g.strip())
    answer_acc = correct_answers / len(predictions['answers'])
    correct_grounding = sum(1 for p, g in zip(predictions['bboxes'], golds['bboxes']) if p == g)
    grounding_acc = correct_grounding / len(predictions['bboxes'])
    return {'answer accuracy': answer_acc, 'grounding accuracy': grounding_acc}

Common pitfalls

• Multiple valid grounding solutions exist for certain chart types (e.g., stacked bar charts), leading to annotator disagreement and ambiguous gold labels.
• Errors from source datasets (ChartQA/ChartQA-PoT) and pipeline failures (heuristic/PoT/GPT stages) can carry over into the benchmark.
• Grounding annotations are strictly tied to data values or structural elements; models must output exact matches to pass validation.

Evidence (verbatim from paper)

Unlike prior datasets that focus solely on answer accuracy, RefChartQA ensures models justify their predictions by linking answers to relevant chart elements. The resulting combined RefChartQA consists of 73,702 image-question-grounding pairs, with 55,789 for training, 6,223 for validation, and 11,690 for testing.

Citation

@misc{vogel2025refchartqa,
  title={RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning},
  author={Vogel et al. (2025)},
  year={2025},
  note={arXiv:2503.23131}
}

• arXiv: 2503.23131