Legal Bench Eval

legal-bench-eval

Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval — Maurya et al. (2025) (arXiv:2509.00141, 2025)

What this evaluates

Evaluates the ability of state-space models (Mamba/SSD-Mamba) and transformers to perform statutory classification and case law retrieval on long-context legal documents. It probes how well models capture fine-grained semantic distinctions and maintain global coherence over thousands of tokens while balancing accuracy with computational throughput.

Datasets

• SCOTUS — total ?; splits: test (-1)
• ILDC — total ?; splits: test (-1)
• ECtHR — total ?; splits: test (-1)
• EUR-Lex — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels out of total instances.
• Micro-F1 — range: [0, 1]
  • F1 score computed globally by counting total true positives, false negatives, and false positives across all classes.
• Macro-F1 — range: [0, 1]
  • F1 score computed per class and then averaged, giving equal weight to each category regardless of frequency.
• Recall@10 — range: [0, 1]
  • Proportion of queries where at least one relevant document appears in the top-10 retrieved results.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality with logarithmic position discounting.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank: average of 1/rank of the first relevant document for each query.
• MAP — range: [0, 1]
  • Mean Average Precision: average of precision values at each rank where a relevant document is retrieved.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring binary classification separability.

Input / output format

Input: Legal documents (statutes or case law) provided as raw text sequences, ranging from moderate to extremely long context lengths.

Output: For classification: predicted category label(s). For retrieval: ranked list of candidate documents or relevance scores.

Scoring recipe

def compute_metrics(predictions, gold, k=10):
    # Classification
    accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    # F1 scores computed via standard confusion matrix aggregation (micro/macro)
    # Retrieval
    recall_k = sum(1 for p in predictions if any(g in p[:k] for g in gold)) / len(gold)
    # nDCG@k, MRR, MAP computed using standard ranking formulas with position discounting
    return {'accuracy': accuracy, 'recall@10': recall_k, 'nDCG@10': ndcg, 'MRR': mrr}

Common pitfalls

• Confusing micro-averaged F1 with macro-averaged F1, which can mask performance on imbalanced legal categories.
• Comparing accuracy without accounting for context length limits or throughput differences, leading to unfair efficiency assessments.
• Treating Recall@10 and nDCG@10 as equivalent; Recall measures coverage while nDCG penalizes rank order errors.

Evidence (verbatim from paper)

On structured inputs such as SCOTUS issue classification (Table[IV]), DeBERTa achieves the strongest overall performance (Micro-F1 83.8, Accuracy 84.0), with Longformer close behind. Similarly, in ILDC retrieval (Table[VI]), both Longformer and DeBERTa outperform Mamba on Recall@10 and nDCG@10.

Citation

@misc{maurya2025scalinglegalai,
  title={Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval},
  author={Maurya et al. (2025)},
  year={2025},
  note={arXiv:2509.00141}
}

• arXiv: 2509.00141