Mlsum Eval

mlsum-eval

MLSUM: The Multilingual Summarization Corpus — Scialom et al. (2020) (arXiv:2004.14900, 2020)

What this evaluates

Evaluates abstractive text summarization models across multiple languages (French, German, Spanish, Russian, Turkish) to measure generation quality and investigate cross-lingual performance gaps and model biases.

Datasets

• MLSUM — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/recitalAI/MLSUM

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Recall-oriented overlap of the longest common subsequence between reference and generated summaries, normalized by reference length.
• METEOR — range: [0, 1]
  • Metric evaluating alignment between generated and reference summaries based on exact, stem, synonym, and paraphrase matches, with penalties for fragmentation.

Input / output format

Input: Source news article text.

Output: Generated summary text.

Scoring recipe

def compute_rouge_l(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        lcs_len = longest_common_subsequence_length(pred, ref)
        recall = lcs_len / len(ref)
        precision = lcs_len / len(pred)
        if recall + precision > 0:
            f_measure = 2 * (recall * precision) / (recall + precision)
        else:
            f_measure = 0.0
        scores.append(f_measure)
    return sum(scores) / len(scores)

Common pitfalls

• ROUGE scores can be artificially inflated or deflated by morphological differences between languages, making direct cross-lingual comparisons misleading without normalization.
• TextRank baselines exhibit a strong English bias, performing poorly on other languages despite being unsupervised, which can skew comparative analyses if not accounted for.
• Oracle extractive performance varies by language not just due to abstractiveness but because relevant information may be more spread across sentences in some languages (e.g., French/Spanish vs. German).

Evidence (verbatim from paper)

Turning to the observed results, we report in Table 2 the ROUGE-L and METEOR scores obtained by each model for all languages. We note that the overall order of systems (for each language) is preserved when using either metric (modulo some swaps between Lead_3 and Pointer Generator, but with relatively close scores).

Citation

@misc{scialom2020mlsum,
  title={MLSUM: The Multilingual Summarization Corpus},
  author={Scialom et al. (2020)},
  year={2020},
  note={arXiv:2004.14900}
}

• arXiv: 2004.14900