legalbench-eval
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models — Guha et al. (2023) (arXiv:2308.11462, 2023)
What this evaluates
This benchmark probes large language models' ability to perform diverse, real-world legal reasoning tasks, including rule-recall, issue-spotting, rule-application, interpretation, and rhetorical understanding. It evaluates how well models can apply legal frameworks, classify contractual clauses, and answer questions based on statutory or case law text.
Datasets
Metrics
accuracy (primary) — range: [0, 1]
- Calculated as the proportion of correctly predicted labels or exact matches for generated outputs divided by the total number of instances in a task. For classification tasks, it measures the percentage of correct binary or multi-class predictions.
Input / output format
Input: Task instructions, few-shot demonstrations, and a specific legal context (e.g., contract clause, court opinion excerpt, statutory text, or privacy policy) followed by a question or classification prompt.
Output: Model must produce a classification label (binary/multi-class), a multiple-choice option, or a generated text/citation string.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
Common pitfalls
- Context window constraints initially limited tasks to short documents, excluding long-form legal reasoning.
- Tasks are strictly objective and do not evaluate reasoning where 'reasonable minds may differ' or involve degrees of correctness.
- Benchmark is heavily skewed toward US law, contracts, and English, limiting generalizability to other jurisdictions or legal domains.
- Evaluates IRAC components (Issue, Rule, Application, Conclusion) independently rather than in multi-hop, sequential legal analysis.
Evidence (verbatim from paper)
Appendix F discusses each task in detail, providing a description of the reasoning that each task evaluates, how task data was constructed, task examples, and evaluation protocols. LEGALBENCH tasks also span different formats: multiple-choice questions (35 tasks), open-generation (7 tasks), binary classification (112 tasks), and multi-class/multi-label classification (8 tasks).
Citation
@misc{guha2023legalbench,
title={LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models},
author={Guha et al. (2023)},
year={2023},
note={arXiv:2308.11462}
}
1---2name: legalbench-eval3description: This benchmark probes large language models' ability to perform diverse, real-world legal reasoning tasks, including rule-recall, issue-spotting, rule-application, interpretation, and rhetorical understanding. It evaluates how well models can apply legal frameworks, classify contractual clauses, and answer questions based on statutory or case law text. Use when the user wants to benchmark on LegalBench, or asks about evaluating this task. Reports accuracy.4---56# legalbench-eval78> LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models — Guha et al. (2023) (arXiv:2308.11462, 2023)910## What this evaluates1112This benchmark probes large language models' ability to perform diverse, real-world legal reasoning tasks, including rule-recall, issue-spotting, rule-application, interpretation, and rhetorical understanding. It evaluates how well models can apply legal frameworks, classify contractual clauses, and answer questions based on statutory or case law text.1314## Datasets1516- **LegalBench** — total ?; splits: test (-1); repo https://github.com/HazyResearch/legalbench1718## Metrics1920- `accuracy` **(primary)** — range: [0, 1]21 - Calculated as the proportion of correctly predicted labels or exact matches for generated outputs divided by the total number of instances in a task. For classification tasks, it measures the percentage of correct binary or multi-class predictions.2223## Input / output format2425**Input**: Task instructions, few-shot demonstrations, and a specific legal context (e.g., contract clause, court opinion excerpt, statutory text, or privacy policy) followed by a question or classification prompt.2627**Output**: Model must produce a classification label (binary/multi-class), a multiple-choice option, or a generated text/citation string.2829## Scoring recipe3031```python32def compute_accuracy(predictions, gold_labels):33 correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)34 return correct / len(gold_labels)35```3637## Common pitfalls3839- Context window constraints initially limited tasks to short documents, excluding long-form legal reasoning.40- Tasks are strictly objective and do not evaluate reasoning where 'reasonable minds may differ' or involve degrees of correctness.41- Benchmark is heavily skewed toward US law, contracts, and English, limiting generalizability to other jurisdictions or legal domains.42- Evaluates IRAC components (Issue, Rule, Application, Conclusion) independently rather than in multi-hop, sequential legal analysis.4344## Evidence (verbatim from paper)4546> Appendix F discusses each task in detail, providing a description of the reasoning that each task evaluates, how task data was constructed, task examples, and evaluation protocols. LEGALBENCH tasks also span different formats: multiple-choice questions (35 tasks), open-generation (7 tasks), binary classification (112 tasks), and multi-class/multi-label classification (8 tasks).4748## Citation4950```bibtex51@misc{guha2023legalbench,52 title={LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models},53 author={Guha et al. (2023)},54 year={2023},55 note={arXiv:2308.11462}56}57```5859- arXiv: 2308.11462
qhjqhj00