Lextreme Eval

lextreme-eval

LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain — Niklaus et al. (2023) (arXiv:2301.13126, 2023)

What this evaluates

Evaluates multilingual legal NLP models across text classification and named entity recognition tasks. It probes the ability of models to handle domain-specific legal jargon, long-form documents, and cross-lingual generalization across 24 languages.

Datasets

• LEXTREME — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/JoelNiklaus/LEXTREME

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Unweighted average of recall (or F1) computed independently for each class, ensuring comparability across imbalanced text classification and NER tasks.
• LEXTREME aggregate score — range: [0, 1]
  • Harmonic mean of the dataset aggregate score and the language aggregate score. Each aggregate is computed by successively taking the harmonic mean of macro-F1 scores across languages, configurations, and datasets.

Input / output format

Input: Multilingual legal documents (sequence lengths up to 4096 tokens) for text classification or named entity recognition tasks.

Output: Predicted class labels for classification tasks or token-level entity annotations for NER tasks.

Scoring recipe

def compute_macro_f1(preds, golds):
    recalls = [recall_per_class(c) for c in unique_classes]
    return mean(recalls)

def compute_lextreme_score(all_f1s):
    # Harmonic mean across languages, configs, datasets
    dataset_agg = harmonic_mean(mean(f1s_per_language))
    language_agg = harmonic_mean(mean(f1s_per_dataset))
    return harmonic_mean(dataset_agg, language_agg)

Common pitfalls

• Datasets are highly imbalanced (e.g., BCD-U has a 2% minority class), requiring random oversampling during fine-tuning.
• Mathew's Correlation Coefficient (MCC) is only suitable for text classification, not NER, despite being reported in supplementary materials.
• Long documents require specific hierarchical model variants and tailored sequence lengths (1024–4096 tokens) rather than standard 512-token truncation.

Evidence (verbatim from paper)

We use the macro-F1 score for all datasets to ensure comparability across the entire benchmark, since it can be computed for both text classification and NER tasks. Mathew's Correlation Coefficient (MCC) (Matthews, 1975) is a suitable score for evaluating text classification tasks but its applicability to NER tasks is unclear. For brevity, we do not display additional scores, but more detailed (such as precision and recall, and scores per seed) and additional scores (such as MCC) can be found online on our Weights and Biases project. The final LEXTREME score is computed using the harmonic mean of the dataset and the language aggregate score. We calculate the dataset aggregate by successively taking the harmonic mean of (i) the languages in the configurations (e.g., de,fr,it in SJP), (ii) configurations within datasets (e.g., OTS-UL, OTS-CT in OTS), and (iii) datasets in LEXTREME (BCD, GAM).

Citation

@misc{niklaus2023lextreme,
  title={LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain},
  author={Niklaus et al. (2023)},
  year={2023},
  note={arXiv:2301.13126}
}

• arXiv: 2301.13126