Eur Lex Sum Eval

eur-lex-sum-eval

EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain — Aumiller et al. (2022) (arXiv:2210.13448, 2022)

What this evaluates

This benchmark evaluates long-form, multi- and cross-lingual summarization capabilities in the legal domain. It probes a model's ability to extract or generate concise summaries from lengthy, structurally complex EU legal documents across 24 official EU languages, including cross-lingual transfer scenarios.

Datasets

• EUR-Lex-Sum — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/achouhan93/eur-lex-sum

Metrics

• ROUGE-1 (primary) — range: percent
  • Unigram overlap F1 score between generated and reference summaries. Computed with stemming disabled to preserve cross-lingual comparability.
• ROUGE-2 — range: percent
  • Bigram overlap F1 score between generated and reference summaries. Stemming is disabled.
• ROUGE-L — range: percent
  • Longest common subsequence F1 score between generated and reference summaries. Stemming is disabled.

Input / output format

Input: Long-form legal document text (chunked by paragraph separators if exceeding model context limits) in a target language.

Output: Extractive summary composed of selected paragraphs, or abstractive summary text.

Scoring recipe

def compute_rouge(pred, ref, rouge_type='rouge1', stem=False):
    p_tokens = tokenize(pred, stem=False)
    r_tokens = tokenize(ref, stem=False)
    if rouge_type == 'rouge1':
        return f1_score(ngrams(p_tokens, 1), ngrams(r_tokens, 1))
    elif rouge_type == 'rouge2':
        return f1_score(ngrams(p_tokens, 2), ngrams(r_tokens, 2))
    elif rouge_type == 'rougel':
        return f1_score(longest_common_subsequence(p_tokens, r_tokens))

Common pitfalls

• Summary lengths often exceed standard transformer context windows (512 or 4096 tokens), requiring paragraph-level chunking that can disrupt coherence if not handled carefully.
• Tokenizer and sentence-splitting availability varies significantly across the 24 EU languages, making fair cross-lingual evaluation difficult for underrepresented languages.
• ROUGE is computed with stemming disabled to ensure cross-lingual comparability, which may penalize morphologically rich languages differently than standard implementations.

Evidence (verbatim from paper)

For evaluation, we rely on ROUGE scores (Lin, 2004) with disabled stemming to conserve comparability between languages. We acknowledge that this is not a comprehensive measure and has distinctive shortcomings, but works fairly well at the paragraph level, as such units generally preserve both factual consistency and fluency.

Citation

@misc{aumiller2022eurlexsum,
  title={EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain},
  author={Aumiller et al. (2022)},
  year={2022},
  note={arXiv:2210.13448}
}

• arXiv: 2210.13448