Wikilingua Eval

wikilingua-eval

WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization — Ladhak et al. (2020) (arXiv:2010.03093, 2020)

What this evaluates

This benchmark evaluates cross-lingual abstractive summarization, specifically the ability of models to generate coherent English summaries from articles written in other languages. It probes how well systems can handle translation and summarization jointly, either through direct cross-lingual fine-tuning or two-step pipeline approaches.

Datasets

• WikiLingua — total 141457; splits: train (-1), dev (-1), test (-1); repo https://github.com/esdurmus/Wikilingua

Metrics

• ROUGE-1 F1 — range: [0, 1]
  • F1 score computed over unigram overlaps between generated summary and reference summary.
• ROUGE-2 F1 — range: [0, 1]
  • F1 score computed over bigram overlaps between generated summary and reference summary.
• ROUGE-L F1 (primary) — range: [0, 1]
  • F1 score based on the longest common subsequence (LCS) between generated and reference summaries, capturing sentence-level fluency and structure.

Input / output format

Input: Source language article tokenized into subwords, with a special separator token between each sentence and a language indicator token appended at the end.

Output: Target language summary tokenized into subwords, with a language indicator token prepended at the beginning to prime the decoder.

Scoring recipe

def compute_rouge_f1(predictions, references, ngram=1):
    scores = []
    for pred, ref in zip(predictions, references):
        rouge = rouge_score.rouge_scores(ref, pred, ngrams=ngram)
        scores.append(rouge[f'rouge{ngram}']['fmeasure'])
    return sum(scores) / len(scores)

Common pitfalls

• Topic leakage must be prevented: English test articles must share topics with non-English test articles to ensure consistent evaluation across languages.
• Translation error isolation: Two-step baselines (Sum-Trans, Trans-Sum) require an oracle system (Trans-Sum-G) with gold translations to accurately measure performance drops caused by machine translation errors.
• Statistical validation: Raw score differences must be validated with Welch's t-test (p < 0.01) to claim significant improvements over baselines.

Evidence (verbatim from paper)

The numbers correspond to ROUGE-1/ROUGE-2/ROUGE-L F1 scores respectively. † indicates where ROUGE-L F1 is significantly better than all baselines, and ‡ indicates where ROUGE-L F1 is significantly better than all baselines except Trans-Sum-R. We use Welch's t-test, and use p < 0.01 to assess significance.

Citation

@misc{ladhak2020wikilingua,
  title={WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization},
  author={Ladhak et al. (2020)},
  year={2020},
  note={arXiv:2010.03093}
}

• arXiv: 2010.03093