Multi Eurlex Eval

multi-eurlex-eval

MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer — Chalkidis et al. (2021) (arXiv:2109.00904, 2021)

What this evaluates

This benchmark evaluates zero-shot cross-lingual transfer and multi-label classification capabilities on legal documents. It probes how well models trained in one language generalize to others, while handling highly skewed label distributions and temporal concept drift across 23 EU languages.

Datasets

• MultiEURLEX — total 65000; splits: chronological (-1); HF multi_eurlex; repo https://github.com/nlpaueb/multi-eurlex

Metrics

• mean R-Precision (mrp) (primary) — range: percent
  • For each document, the model ranks the predicted labels by decreasing confidence. Precision@$k$ is computed where $k$ is the number of gold labels for that document. The metric is averaged over all documents.

Input / output format

Input: Raw legal document text in one of 23 EU official languages.

Output: A ranked list of predicted labels ordered by decreasing confidence score.

Scoring recipe

def compute_mrp(predictions, gold_labels):
    total_prec = 0.0
    for preds, gold in zip(predictions, gold_labels):
        k = len(gold)
        if k == 0: continue
        top_k_labels = [label for label, _ in preds[:k]]
        hits = sum(1 for l in top_k_labels if l in gold)
        total_prec += hits / k
    return (total_prec / len(predictions)) * 100

Common pitfalls

• Using random train/test splits instead of chronological splits, which ignores temporal concept drift in legal topics and yields artificially inflated results.
• Applying standard accuracy or macro-F1 instead of mean R-Precision, which fails to account for the highly skewed (long-tail) multi-label distribution.
• Assuming end-to-end fine-tuning in a single source language preserves cross-lingual knowledge, as it actually causes catastrophic forgetting for target languages.

Evidence (verbatim from paper)

Following Chalkidis et al. (2019, 2020a), we report mean R-Precision (mrp) Manning et al. (2009). That is, for each document, the model ranks the labels it selects by decreasing confidence, and we compute Precision@$k$, where $k$ is the document’s number of gold labels; we then average over documents. For all experiments, we use the chronological data split and report the average across three runs.

Citation

@misc{chalkidis2021multieurlex,
  title={MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer},
  author={Chalkidis et al. (2021)},
  year={2021},
  note={arXiv:2109.00904}
}

• arXiv: 2109.00904