Chart QA Eval

chart-qa-eval

BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning — Masry et al. (2025) (arXiv:2508.09804, 2025)

What this evaluates

Evaluates vision-language models on chart question answering, probing their ability to extract numerical data, perform visual interpolation, and reason over diverse chart types. It measures both direct data retrieval and complex reasoning capabilities across real-world and synthetic chart distributions.

Datasets

• FigureQA-Sub — total 1000; splits: Val1 (-1), Val2 (-1)
• DVQA-Sub — total 1000; splits: ValE (-1), ValH (-1)
• PlotQA-Sub — total 1000; splits: T1 (-1), T2 (-1)
• ChartQA — total ?; splits: aug (-1), hum (-1)
• CharXiv — total ?; splits: Reas. (-1), Des. (-1)

Metrics

• exact accuracy (primary) — range: percent
  • 1 if the model's predicted answer exactly matches the ground truth answer string, 0 otherwise. Applied to FigureQA-Sub and DVQA-Sub.
• relaxed accuracy (primary) — range: percent
  • 1 if the predicted answer falls within a predefined numerical tolerance of the ground truth (accounting for rounding, formatting, or unit differences), 0 otherwise. Applied to PlotQA-Sub and ChartQA.
• GPT-4o LLM-as-judge — range: percent
  • GPT-4o evaluates model outputs against gold answers using a prompt from the original CharXiv paper, returning a binary or scaled score. Applied to CharXiv.

Input / output format

Input: A chart image paired with a natural language question.

Output: A text response containing the answer, which may be a numerical value, a short phrase, or a step-by-step reasoning chain.

Scoring recipe

def compute_accuracy(predictions, golds, metric_type='exact'):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if metric_type == 'exact':
            if str(pred).strip().lower() == str(gold).strip().lower():
                correct += 1
        elif metric_type == 'relaxed':
            if is_numerically_close(pred, gold, tol=0.01):
                correct += 1
        elif metric_type == 'llm_judge':
            score = call_gpt4o_judge(pred, gold, prompt=charxiv_prompt)
            if score == 1: correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Synthetic benchmarks (FigureQA, DVQA, PlotQA) often lack explicit numerical labels on chart elements, forcing models to visually interpolate values rather than read them directly. Prior evaluations frequently overlook this distinction, leading to unfair comparisons.
• Full-scale synthetic splits are computationally prohibitive (e.g., PlotQA Test1 has 1.1M pairs). The paper uses 1K subsamples ('-Sub' versions) to reduce cost while preserving diversity, which may not fully represent original distribution performance.
• CharXiv evaluation relies on an external LLM (GPT-4o) with a specific prompt, introducing potential prompt-sensitivity and non-determinism compared to exact/relaxed accuracy metrics.

Evidence (verbatim from paper)

Evaluation Metrics. For our evaluations, we use exact accuracy as the metric for FigureQA-Sub and DVQA-Sub, while we use relaxed accuracy (Methani et al., 2020; Masry et al., 2022) for PlotQA-Sub and ChartQA. For CharXiv, we use GPT4o in conjunction with the prompt proposed in the original work (Wang et al., 2024) for evaluation.

Citation

@misc{masry2025bigchartsr1,
  title={BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning},
  author={Masry et al. (2025)},
  year={2025},
  note={arXiv:2508.09804}
}

• arXiv: 2508.09804