Cnndm Eval

cnndm-eval

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

What this evaluates

Evaluates abstractive summarization quality by scoring generated summaries against human-written references and expert rubric-based scores. It measures how well automatic metrics correlate with human judgments across different summarization systems.

Datasets

• CNNDM — total ?; splits: test (-1)

Metrics

• COMET (primary) — range: [0, 1]
  • A neural reference-based metric trained to predict human judgments. Scores are typically normalized to [0, 1].
• COMET-QE — range: [0, 1]
  • A referenceless variant of COMET that uses only the source sentence and hypothesis to predict quality.
• BERTScore — range: [0, 1]
  • A reference-based metric that computes semantic similarity using contextual embeddings from BERT.

Input / output format

Input: News article text.

Output: Generated summary.

Scoring recipe

scores = []
for article, refs, summary in test_subset:
    metric_score = compute_metric(summary, refs)
    human_score = average(expert_scores[summary]) # avg over 4 dims & 3 experts
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])

Common pitfalls

• Including author-written highlights in the reference set, which are often noisy or lack coherence.
• Not applying NLTK detokenization or Stanford CoreNLP truecasing to model outputs before evaluation.

Evidence (verbatim from paper)

Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: COMET ... Ensemble of Metrics: 2.85·COMET+0.26·COMET-QE+0.01·BERTScore ... 100 test articles are annotated with 10 summaries written by humans... Each output summary is evaluated by three experts along four dimensions: coherence, consistency, fluency, and relevance. An instance-level score is computed by averaging scores over all these categories and the three experts.

Citation

@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}

• arXiv: 2112.04139