Vprochart Eval

vprochart-eval

VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning — Huang et al. (2024) (arXiv:2409.01667, 2024)

What this evaluates

Evaluates a model's ability to understand chart visuals and perform multi-step numerical and logical reasoning to answer natural language questions. It specifically probes visual perception alignment and programmatic solution reasoning over structured chart data.

Datasets

• ChartQA — total ?; splits: Human (-1), Augment (-1)
• PlotQA — total ?; splits: D1 (-1), D2 (-1)
• DVQA — total ?; splits: Familiar (-1), Novel (-1)

Metrics

• accuracy (primary) — range: percent
  • Number of correctly predicted answers divided by the total number of questions, multiplied by 100. Answers are typically compared using exact match or normalized string matching.

Input / output format

Input: A chart image paired with a natural language question.

Output: A natural language answer or numerical value.

Scoring recipe

def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if str(p).strip().lower() == str(g).strip().lower())
    return correct / len(gold) * 100

Common pitfalls

• ChartQA-H (Human) and ChartQA-M (Augment) have vastly different difficulty levels due to manual curation versus machine generation; mixing them up or reporting only one skews results.
• PlotQA-D1 is often used for training or in-domain testing, while D2 is a held-out generalization set; reporting only D1 inflates perceived robustness.
• DVQA's Familiar vs Novel split tests generalization to unseen chart types and data distributions, which is frequently overlooked when only aggregate scores are reported.

Evidence (verbatim from paper)

In Table [1] and [2], we present a comprehensive summary of VProChart’s performance on three popular datasets.

Citation

@misc{huang2024vprochart,
  title={VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2409.01667}
}

• arXiv: 2409.01667