Chart Hqa Eval

chart-hqa-eval

Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts — Chen et al. (2025) (arXiv:2503.04095, 2025)

What this evaluates

Evaluates multimodal large language models' ability to perform counterfactual reasoning over chart visualizations. It probes whether models rely on parametric memory or truly understand the visual data when answering questions that contain hypothetical assumptions about the chart.

Datasets

• Chart-HQA — total ?; splits: test (-1)

Metrics

• relaxed accuracy (primary) — range: percent
  • Exact match accuracy with a 5% tolerance on numerical error for numerical answers; exact string matching for categorical/text answers.
• Decline Rate — range: percent
  • Calculated as |Acc_QA - Acc_HQA| / Acc_QA * 100%, measuring the performance drop from factual QA to hypothetical QA.

Input / output format

Input: A chart image and a natural language question containing a hypothetical/counterfactual assumption about the chart data.

Output: A textual or numerical answer corresponding to the question.

Scoring recipe

def compute_relaxed_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if isinstance(gold, (int, float)):
            if abs(float(pred) - float(gold)) / max(abs(float(gold)), 1e-9) <= 0.05:
                correct += 1
        else:
            if str(pred).strip().lower() == str(gold).strip().lower():
                correct += 1
    return correct / len(golds) * 100

def compute_decline_rate(acc_qa, acc_hqa):
    return abs(acc_qa - acc_hqa) / acc_qa * 100

Common pitfalls

• Models frequently ignore the hypothetical assumption and answer based on the original chart data or pre-trained parametric knowledge, leading to high factual accuracy but poor HQA performance.
• The 5% numerical tolerance applies only to numerical answers; applying it to categorical or boolean answers will incorrectly penalize valid responses.
• Evaluating closed-source models via APIs may introduce latency or rate-limiting artifacts not present in open-source re-implementations, affecting fair comparison.

Evidence (verbatim from paper)

Specifically, we choose the relaxed accuracy used in ChartQA as the evaluation metric, which means exact match accuracy with 5% tolerance on numerical error is used to report all QA results. In addition, we compute the decline rate to measure the performance difference of models between ChartQA and Chart-HQA, which is calculated as follows: |Acc_QA - Acc_HQA| / Acc_QA * 100%.

Citation

@misc{chen2025charthqa,
  title={Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2503.04095}
}

• arXiv: 2503.04095