Orangesum Eval

orangesum-eval

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model — Moussa Kamal Eddine et al. (2020) (arXiv:2010.12321, 2020)

What this evaluates

Evaluates abstractive French summarization quality by measuring lexical overlap, semantic similarity, and human judgments of accuracy, informativeness, and fluency.

Datasets

• OrangeSum — total ?; splits: abstract (-1), title (-1)

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • F-measure based on the longest common subsequence between generated and reference summaries.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of candidate and reference tokens, optionally rescaled.
• Human Evaluation Score — range: percent
  • Percentage of times a model's summary was chosen as best minus percentage chosen as worst in Best-Worst Scaling across accuracy, informativeness, and fluency.

Input / output format

Input: French news document text

Output: Generated summary text (abstract or title length)

Scoring recipe

rouge_l = compute_rouge_l(generated, gold)
bertscore = compute_bertscore(generated, gold)
best_count = 0; worst_count = 0; total = 0
for doc, pair in summary_pairs:
    choice = human_judge(doc, pair.sys1, pair.sys2, ['accuracy', 'informativeness', 'fluency'])
    if choice == pair.sys1: best_count += 1
    elif choice == pair.sys2: worst_count += 1
    total += 1
human_score = (best_count / total) * 100 - (worst_count / total) * 100

Common pitfalls

• ROUGE scores poorly correlate with abstractive quality and can be misleading for models that generate irrelevant novel words.
• Human evaluators may prefer model outputs over ground truth if references contain external information (e.g., dates) not present in the source document.
• BERTScore reports two values (with/without rescaling); must specify which convention is used.

Evidence (verbatim from paper)

We classically report ROUGE-1, ROUGE-2 and ROUGE-L scores (Lin, 2004) in Table 6. However, since ROUGE is limited to capturing n-gram overlap, which is poorly suited to the abstractive summarization setting, we also report BERTScore scores. BERTScore (Zhang et al., 2019) is a recently introduced metric that leverages the contextual representations of the candidate and gold sentences.

Citation

@misc{moussa2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Moussa Kamal Eddine et al. (2020)},
  year={2020},
  note={arXiv:2010.12321}
}

• arXiv: 2010.12321