Xwikis Summarisation Eval

xwikis-summarisation-eval

Models and Datasets for Cross-Lingual Summarisation — Perez-Beltrachini et al. (2022) (arXiv:2202.09583, 2022)

What this evaluates

This benchmark evaluates the ability of multilingual and cross-lingual models to generate accurate English summaries from source documents in German, French, and Czech. It probes supervised, zero-shot, and few-shot cross-lingual transfer capabilities, as well as model robustness on out-of-domain news text.

Datasets

• XWikis — total ?; splits: train (-1), val (-1), test (-1)
• D_en→en — total 300000; splits: train (270000), val (15000), test (15000)
• Voxeurop — total 2666; splits: test (2000), adaptation (666)

Metrics

• ROUGE-L recall (primary) — range: percent
  • Standard ROUGE-L recall computed over the generated summary against the reference summary, measuring the recall of the longest common subsequence. Values in the paper's tables are reported as percentages (0-100).

Input / output format

Input: Source document text in a source language (German, French, Czech, or English), optionally truncated or extracted to a 600-token budget using paragraph-based LEXRANK ranking to handle long documents.

Output: A generated English summary text corresponding to the input document.

Scoring recipe

def compute_rouge_l_recall(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        lcs_len = longest_common_subsequence_length(pred, ref)
        recall = lcs_len / len(ref) if len(ref) > 0 else 0.0
        scores.append(recall * 100)
    return sum(scores) / len(scores)

Common pitfalls

• Evaluating the extractive paragraph selection step using the cross-lingual reference instead of the monolingual reference summary, which the authors explicitly warn against.
• Confusing the XWikis-comparable split (used for training/validation) with the XWikis-parallel split (used for final evaluation).
• Ignoring the 600-token input budget constraint, which significantly alters the effective input length compared to full Wikipedia articles and impacts model performance.

Evidence (verbatim from paper)

Table 5 reports ROUGE-L recall of the input against the reference summary (note that to measure this we take the monolingual summary associated with the document rather than the cross-lingual one). As can be seen, the extractive step reduces the document to a manageable size without sacrificing too much content.

Citation

@misc{perezbeltrachini2022models,
  title={Models and Datasets for Cross-Lingual Summarisation},
  author={Perez-Beltrachini et al. (2022)},
  year={2022},
  note={arXiv:2202.09583}
}

• arXiv: 2202.09583