Chartdiff Eval

chartdiff-eval

ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts — Ye (2026) (arXiv:2603.28902, 2026)

What this evaluates

This benchmark evaluates a model's ability to perform cross-chart comparative reasoning by generating natural language summaries that identify and explain differences in trends, fluctuations, and anomalies between pairs of charts. It probes vision-language models on their capacity to synthesize visual information from multiple plots into coherent, human-aligned textual descriptions.

Datasets

• ChartDiff — total 8541; splits: train (-1), test (-1)

Metrics

• ROUGE — range: percent
  • Standard lexical-overlap metric measuring n-gram overlap between generated and reference summaries. Reported as ROUGE-1, ROUGE-2, and ROUGE-L.
• GPT Score (primary) — range: [1, 5]
  • Model-based evaluation using GPT-5.4 as a judge with a predefined grading prompt to assess summary quality. Scores range from 1 to 5.

Input / output format

Input: Two charts concatenated horizontally into a single combined image, accompanied by a prompt instructing the model to generate a comparison summary.

Output: A natural language text summary describing the differences between the two charts.

Scoring recipe

def score_rouge(reference, prediction):
    return rouge_score(reference, prediction, rouge_types=['rouge1', 'rouge2', 'rougeL'])

def score_gpt(reference, prediction):
    prompt = f'Grade the quality of this summary against the reference: Pred="{prediction}", Ref="{reference}".'
    response = call_llm('gpt-5.4', prompt)
    return extract_numeric_score(response)

Common pitfalls

• High ROUGE scores indicate strong lexical overlap but do not guarantee human-aligned quality or accurate difference detection.
• Pipeline-based extract-then-compare methods are highly sensitive to chart type (e.g., failing on pie charts) and plotting library rendering.
• Multi-series charts consistently yield lower scores across all model families compared to single-series charts.

Evidence (verbatim from paper)

We adopt two complementary evaluation metrics: ROUGE. We use ROUGE as a standard lexical-overlap metric to measure similarity between generated summaries and reference annotations. GPT Score. We further employ a model-based evaluation metric, GPT Score using GPT-5.4 as the judge model to assess the quality of generated summaries with a predefined grading prompt (Figure 16 and Figure 17).

Citation

@misc{ye2026chartdiff,
  title={ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts},
  author={Ye (2026)},
  year={2026},
  note={arXiv:2603.28902}
}

• arXiv: 2603.28902