qhjqhj00 legal-text-classification-eval
One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support — Stern et al. (2023) (arXiv:2306.09237, 2023)
What this evaluates
This benchmark assesses models on hierarchical legal text classification tasks (LAP, JP, CP) across three Swiss languages. It tests domain-specific classification accuracy and robustness to long legal documents and multilingual inputs.
Datasets
- Legal Classification (LAP, JP, CP) — total ?; splits: train (-1), val (1000), test (-1)
Metrics
Hierarchical Macro-F1(primary) — range: [0, 1]- Harmonic mean of macro-averaged F1 scores aggregated across languages, configurations, and datasets to penalize outlier low scores and promote fairness.
Input / output format
Input: Legal documents (up to 4096 tokens) with hierarchical class labels, provided in German, French, or Italian.
Output: Predicted hierarchical class label(s) for each document.
Scoring recipe
def score_classification(predictions, golds, languages, configs, datasets):
f1_scores = []
for lang in languages:
for cfg in configs:
for ds in datasets:
mask = (lang & cfg & ds)
f1_scores.append(macro_f1(predictions[mask], golds[mask]))
return harmonic_mean(f1_scores)
Common pitfalls
- Zero-shot evaluation uses only one instruction and example, which may underperform compared to few-shot setups.
- Truncation at 4096 tokens can discard relevant legal context, artificially lowering classification F1.
Evidence (verbatim from paper)
For our TC tasks, namely LAP, JP, and CP, we adopted the LEXTREME benchmark setup [10], namely hierarchical aggregation of macro-averaged F1 scores using harmonic mean for fairness (the harmonic mean is biased more towards lower scores than the geometric or arithmetic mean).
Citation
@misc{stern2023legal,
title={One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support},
author={Stern et al. (2023)},
year={2023},
note={arXiv:2306.09237}
}
- arXiv: 2306.09237