Mleb Eval

mleb-eval

The Massive Legal Embedding Benchmark (MLEB) — Umar Butler, Abdur-Rahman Butler, Adrian Lucas Malec (2025) (arXiv:2510.19365, 2025)

What this evaluates

This benchmark evaluates the capability of embedding models to perform legal information retrieval across diverse jurisdictions, document types, and legal tasks. It probes how well models understand judicial reasoning, regulatory interpretation, and multinational contract analysis compared to general-purpose IR models.

Datasets

• MLEB — total ?; splits: test (-1); repo https://github.com/isaacus-dev/mleb

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list of retrieved documents by comparing it to a graded relevance judgment, discounting gains logarithmically by position and normalizing by the ideal DCG.

Input / output format

Input: Queries paired with legal document collections (cases, legislation, contracts, regulatory guidance) across six jurisdictions.

Output: A ranked list of retrieved documents for each query, evaluated at the top 10 positions.

Scoring recipe

def compute_ndcg_at_10(gold_relevance, predicted_ranking):
    k = 10
    dcg = sum(gold_relevance[i] / math.log2(rank + 2) for rank, i in enumerate(predicted_ranking[:k]))
    ideal_rels = sorted(gold_relevance, reverse=True)
    idcg = sum(ideal_rels[i] / math.log2(i + 2) for i in range(k))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Data leakage risk: Some commercial models (Voyage AI, Jina, Google) opt users into data sharing by default, potentially contaminating training data with benchmark queries.
• Cohere models are excluded from evaluation due to their terms of service forbidding benchmarking, which may skew comparative results.
• General multilingual IR performance does not correlate with legal IR performance; models optimized for general tasks may underperform domain-adapted ones.

Evidence (verbatim from paper)

As of 21 October 2025, Isaacus’ Kanon 2 Embedder legal embedding model ranks first on MLEB out of 20 other models, with an NDCG@10 score of 86.03, followed by Voyage 3 Large at 85.71 and Voyage 3.5 at 84.07. The full results of the benchmark are presented below. All scores are NDCG@10 scores. We report both the task average (i.e., by evaluation set) and the domain average.

Citation

@misc{butler2025mleb,
  title={The Massive Legal Embedding Benchmark (MLEB)},
  author={Umar Butler, Abdur-Rahman Butler, Adrian Lucas Malec (2025)},
  year={2025},
  note={arXiv:2510.19365}
}

• arXiv: 2510.19365