# Legal Lens Eval

> legal-lens-eval

- Skill: `qhjqhj00/legal-lens-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/legal-lens-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/legal-lens-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/legal-lens-eval

---


# legal-lens-eval

> LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text — Bernsohn et al. (2024) (arXiv:2402.04335, 2024)

## What this evaluates

This benchmark evaluates language models on identifying legal violations and associating victims in unstructured legal text. It probes two core capabilities: named entity recognition for specific causes of action and natural language inference for linking victims to legal claims across different legal domains.

## Datasets

- **LegalLens NER Dataset** — total ?; splits: train (-1), test (-1); repo https://github.com/darrow-labs/LegalLens
- **LegalLens NLI Dataset** — total ?; splits: train (-1), test (-1); repo https://github.com/darrow-labs/LegalLens

## Metrics

- `Macro F1` **(primary)** — range: [0, 1]
  - Standard macro-averaged F1 score computed across all classes (Causes of Action for NER) or legal domains (for NLI). It calculates the unweighted mean of the F1 scores for each individual class.

## Input / output format

**Input**: Unstructured text snippets or news articles containing legal claims, provided as raw text strings.

**Output**: For NER: predicted Cause of Action (CoA) labels. For NLI: victim association labels. LLMs are guided by a structured Input/Output prompt format specifying the expected prediction structure.

## Scoring recipe

```python
def macro_f1(predictions, golds, classes):
    f1_scores = []
    for cls in classes:
        y_true = [1 if g == cls else 0 for g in golds]
        y_pred = [1 if p == cls else 0 for p in predictions]
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Data leakage can occur if Causes of Action (CoAs) or legal domains overlap between splits; the protocol explicitly requires excluding training CoAs/domains from the test set.
- Few-shot evaluation for closed-source models uses 9 random training samples and temperature 0.7, introducing high variance; results must be averaged over 5 API calls to match reported scores.
- Exact dataset sizes and split counts are not disclosed, making exact replication of the leave-one-out NLI splits or CoA-based NER splits difficult without the source code.

## Evidence (verbatim from paper)

> Our dataset is categorized by Cause of Action (CoA). CoA refers to a set of facts or legal reasons that justify the right to sue or seek legal remedy in a court of law. Due to the potential overlap and similarities between different CoAs, there’s a risk of data leakage when training models. To mitigate this, we adopted a strategy where CoAs present in the training set were excluded from the test set. ... Table 3: Macro F1 evaluation of various model architectures for the NLI task across different legal entities.

## Citation

```bibtex
@misc{berns2024legallens,
  title={LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text},
  author={Bernsohn et al. (2024)},
  year={2024},
  note={arXiv:2402.04335}
}
```

- arXiv: 2402.04335