qhjqhj00 legal-information-retrieval-eval
One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support — Stern et al. (2023) (arXiv:2306.09237, 2023)
What this evaluates
This benchmark evaluates information retrieval systems on Swiss legal rulings and legislation. It tests the ability to rank relevant legal documents against long, multilingual queries and corpora.
Datasets
- Legal Information Retrieval — total ?; splits: train (-1), val (-1), test (-1)
Metrics
NDCG(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain measures ranking quality by weighting relevance scores logarithmically by position to reward higher-ranked relevant documents.
Capped Recall@k— range: [0, 1]- Measures the proportion of relevant documents retrieved within the top-k results, capped at a maximum threshold to handle varying ground-truth sizes.
Input / output format
Input: Multilingual legal queries (avg. 847 words) and a corpus of Swiss legal documents (rulings/legislation, avg. 4K-7K words).
Output: Ranked list of document IDs or scores corresponding to the query.
Scoring recipe
def score_ir(predictions, golds, k_values=[1, 5, 10]):
ndcg_scores = []
recall_scores = []
for q in queries:
pred_ranking = predictions[q]
rel_labels = golds[q]
ndcg_scores.append(ndcg_at_k(pred_ranking, rel_labels))
recall_scores.append(capped_recall_at_k(pred_ranking, rel_labels, k_values))
return {'NDCG': mean(ndcg_scores), 'Capped_Recall@k': mean(recall_scores)}
Common pitfalls
- BM25 is used for scalability but lacks contextual/multilingual understanding, potentially skewing baselines.
- Cross-encoder models were excluded due to computational cost on long documents, limiting state-of-the-art comparison.
Evidence (verbatim from paper)
We evaluate models with Normalized Discounted Cumulative Gain (NDCG) [67] and Capped Recall@k [11].
Citation
@misc{stern2023legal,
title={One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support},
author={Stern et al. (2023)},
year={2023},
note={arXiv:2306.09237}
}
- arXiv: 2306.09237