qhjqhj00 legal-lens-eval
LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text — Bernsohn et al. (2024) (arXiv:2402.04335, 2024)
What this evaluates
This benchmark evaluates language models on identifying legal violations and associating victims in unstructured legal text. It probes two core capabilities: named entity recognition for specific causes of action and natural language inference for linking victims to legal claims across different legal domains.
Datasets
- LegalLens NER Dataset — total ?; splits: train (-1), test (-1); repo https://github.com/darrow-labs/LegalLens
- LegalLens NLI Dataset — total ?; splits: train (-1), test (-1); repo https://github.com/darrow-labs/LegalLens
Metrics
Macro F1(primary) — range: [0, 1]- Standard macro-averaged F1 score computed across all classes (Causes of Action for NER) or legal domains (for NLI). It calculates the unweighted mean of the F1 scores for each individual class.
Input / output format
Input: Unstructured text snippets or news articles containing legal claims, provided as raw text strings.
Output: For NER: predicted Cause of Action (CoA) labels. For NLI: victim association labels. LLMs are guided by a structured Input/Output prompt format specifying the expected prediction structure.
Scoring recipe
def macro_f1(predictions, golds, classes):
f1_scores = []
for cls in classes:
y_true = [1 if g == cls else 0 for g in golds]
y_pred = [1 if p == cls else 0 for p in predictions]
tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
f1_scores.append(f1)
return sum(f1_scores) / len(f1_scores)
Common pitfalls
- Data leakage can occur if Causes of Action (CoAs) or legal domains overlap between splits; the protocol explicitly requires excluding training CoAs/domains from the test set.
- Few-shot evaluation for closed-source models uses 9 random training samples and temperature 0.7, introducing high variance; results must be averaged over 5 API calls to match reported scores.
- Exact dataset sizes and split counts are not disclosed, making exact replication of the leave-one-out NLI splits or CoA-based NER splits difficult without the source code.
Evidence (verbatim from paper)
Our dataset is categorized by Cause of Action (CoA). CoA refers to a set of facts or legal reasons that justify the right to sue or seek legal remedy in a court of law. Due to the potential overlap and similarities between different CoAs, there’s a risk of data leakage when training models. To mitigate this, we adopted a strategy where CoAs present in the training set were excluded from the test set. ... Table 3: Macro F1 evaluation of various model architectures for the NLI task across different legal entities.
Citation
@misc{berns2024legallens,
title={LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text},
author={Bernsohn et al. (2024)},
year={2024},
note={arXiv:2402.04335}
}
- arXiv: 2402.04335