Chartsumm Eval

chartsumm-eval

ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries — Rahman et al. (2023) (arXiv:2304.13620, 2023)

What this evaluates

Evaluates automatic chart-to-text summarization models on their ability to generate accurate, fluent, and informative summaries from chart metadata and data tables. It probes factual correctness, trend capture, and hallucination resistance across short and long summary formats.

Datasets

• ChartSumm — total 84363; splits: train-k (34503), train-s (32985), valid-k (4338), valid-s (4101), test-k (4338), test-s (4098); repo https://github.com/pranonrahman/ChartSumm

Metrics

• BLEU (primary) — range: percent
  • Computes n-gram overlap precision between reference and candidate text, typically reported as a percentage.
• BLEURT — range: other
  • Evaluates fluency and semantic similarity to the reference using a pre-trained language model (base-128 variant).
• Perplexity — range: other
  • Measures how well a probability model predicts a sample; lower values indicate better performance. Computed using a pre-trained GPT-2 model.
• CIDEr — range: other
  • Calculates n-gram overlaps and computes average cosine similarity between candidate and reference sentences to capture semantic richness.
• Content Selection (CS) — range: [0, 1]
  • Measures how closely the generated text matches the reference documents in terms of selected content.

Input / output format

Input: Chart metadata (title, data table flattened by rows, labels) concatenated with a table caption, separated by a token, and prefixed with 'Summarize chart:'.

Output: A natural language summary of the chart (short or long format depending on the split).

Scoring recipe

def score(predictions, references):
    bleu = compute_bleu(references, predictions) * 100
    bleuurt = compute_bleurt(references, predictions)
    ppl = compute_perplexity(predictions, model='gpt2')
    cider = compute_cider(references, predictions)
    cs = compute_content_selection(references, predictions)
    return {'BLEU': bleu, 'BLEURT': bleuurt, 'PPL': ppl, 'CIDEr': cider, 'CS': cs}

Common pitfalls

• Models often generate fluent but factually incorrect summaries or hallucinate irrelevant details (e.g., headquarters location).
• Fine-tuning on one data source (Knoema or Statista) severely hurts generalization to the other source's test set.
• BLEURT and CIDEr are unavailable for non-English languages like Bengali, limiting cross-lingual evaluation to BLEU only.

Evidence (verbatim from paper)

We use five evaluation metrics in our automated evaluation: (i) BLEU (Post, 2018): it uses n-gram overlaps between reference text and machine-generated text to determine similarity score, (ii) BLEURT (Sellam et al., 2020): it evaluates how fluent the candidate is and how well it transfers the reference's meaning (we utilize BLEURT base-128 for our evaluation), (iii) Perplexity: it is a measurement that quantifies how well a probability model predicts a sample (we utilized pre-trained GPT-2 (Radford et al., 2019) to measure perplexity), (iv) CIDEr: (Vedantam et al., 2015) it uses n-gram overlaps and calculates average cosine similarity between the candidate sentence and the reference sentences, to capture the grammatical qualities with richer semantics, (v): Content Selection (CS): it measures how closely the generated text matches the reference documents (Wiseman et al., 2017).

Citation

@misc{rahman2023chartsumm,
  title={ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries},
  author={Rahman et al. (2023)},
  year={2023},
  note={arXiv:2304.13620}
}

• arXiv: 2304.13620