qhjqhj00 polychartqa-eval
Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts — Efat et al. (2026) (arXiv:2604.21344, 2026)
What this evaluates
Evaluates multimodal language models' ability to answer questions that require reasoning across multiple charts or images. It probes visual decomposition, sub-chart localization, and handling of complex multi-visual contexts versus single-chart inputs.
Datasets
- PolyChartQA — total ?; splits: test (-1); repo https://github.com/NRT-D4/PolyChartQA
- MultiChartQA-RQ1 — total 365; splits: test (365)
Metrics
L-Accuracy(primary) — range: [0, 1]- LLM-based accuracy computed by prompting Claude-3.7-Sonnet as a judge to determine if the model's generated answer matches the ground truth.
H-Accuracy— range: [0, 1]- Human-evaluation accuracy where human annotators verify model-predicted answers against ground truth for human-authored questions in a zero-shot setting.
BERTScore— range: [0, 1]- Measures semantic similarity between model predictions and reference answers using BERT embeddings.
Input / output format
Input: Multi-chart images (composite of multiple sub-charts) or single-chart images, paired with a natural language question.
Output: Natural language answer string corresponding to the question.
Scoring recipe
def compute_l_accuracy(pred, gt):
prompt = f'Question: {pred}\nGround Truth: {gt}\nIs the answer correct?'
return 1.0 if llm_judge(prompt) == 'Yes' else 0.0
def compute_bertscore(pred, gt):
return bert_score.compute(predictions=[pred], references=[gt])[0]
def compute_h_accuracy(pred, gt):
return 1.0 if human_evaluator(pred, gt) == 'Correct' else 0.0
Common pitfalls
- Using an LLM judge without verifying alignment with human evaluation can produce misleading L-Accuracy scores.
- Evaluating MultiChartQA-RQ1 requires strict separation of single-chart vs multi-chart input configurations, as the questions are identical across conditions.
- Relying on a single evaluation run may obscure minor variability across model versions or prompting strategies.
Evidence (verbatim from paper)
Evaluation Metrics: We use H-Accuracy (Human-evaluation), L-Accuracy (LLM-based accuracy) introduced by the previous works Pramanick et al. (2024); Liu et al. (2024a); Tang et al. (2025); Wang et al. (2024), and BERTScore Zhang et al. (2020). BERTScore measures semantic similarity between model predictions and references. H-accuracy was obtained through human evaluation of model-predicted answers for human-authored questions in a Zero-shot setting. As human evaluation is difficult to scale, for all other cases, we use L-accuracy by prompting a selected LLM as a judge to assess whether the ground-truth and model-generated answers are similar or not.
Citation
@misc{efat2026beyond,
title={Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts},
author={Efat et al. (2026)},
year={2026},
note={arXiv:2604.21344}
}
- arXiv: 2604.21344