# Legal Bert Eval

> Evaluates domain-adapted BERT models on legal text classification and named entity recognition to measure the impact of further pre-training and hyperparameter tuning strategies. Use when the user wants to benchmark on EURLEX57K, ECHR-CASES, CONTRACTS-NER, or asks about evaluating this task. Reports accuracy, F1.

- Skill: `qhjqhj00/legal-bert-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/legal-bert-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/legal-bert-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/legal-bert-eval

---


# legal-bert-eval

> LEGAL-BERT: The Muppets straight out of Law School — Chalkidis et al. (2020) (arXiv:2010.02559, 2020)

## What this evaluates

Evaluates domain-adapted BERT models on legal text classification and named entity recognition to measure the impact of further pre-training and hyperparameter tuning strategies.

## Datasets

- **EURLEX57K** — total ?; splits: train (-1), val (-1), test (-1)
- **ECHR-CASES** — total ?; splits: train (-1), val (-1), test (-1)
- **CONTRACTS-NER** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy, F1` **(primary)** — range: [0, 1]
  - Standard accuracy and F1 scores for multi-label classification and sequence tagging, as replicated from prior work (Chalkidis et al., 2019).

## Input / output format

**Input**: Raw legal text documents for classification tasks; tokenized text sequences for named entity recognition.

**Output**: Predicted class labels (binary or multi-label) for classification; predicted entity tags for sequence tagging.

## Scoring recipe

```python
# For classification tasks
acc = sum(pred == gold for pred, gold in zip(predictions, gold_labels)) / len(gold_labels)
# For NER tasks
f1 = compute_f1(predictions, gold_labels) # Standard BIO tag matching
# Aggregate across datasets/tasks as needed
```

## Common pitfalls

- Blindly following default BERT hyperparameters (learning rate, epochs, batch size) without domain-specific tuning.
- Using a fixed maximum number of training epochs (e.g., 4) which can cause underfitting on legal datasets; early stopping on validation loss is recommended.

## Evidence (verbatim from paper)

> We evaluate our models on text classification and sequence tagging using three datasets. EURLEX57K (Chalkidis et al., 2019b) is a large-scale multi-label text classification dataset of EU laws, also suitable for few and zero-shot learning. ECHR-CASES (Chalkidis et al., 2019a) contains cases from the European Court of Human Rights (Aletras et al., 2016) and can be used for binary and multi-label text classification. Finally, CONTRACTS-NER (Chalkidis et al., 2017, 2019d) is a dataset for named entity recognition on US contracts consisting of three subsets, contract header, dispute resolution, and lease details. We replicate the experiments of Chalkidis et al. (2019c,a,d) when fine-tuning BERT for all datasets.

## Citation

```bibtex
@misc{chalkidis2020legalbert,
  title={LEGAL-BERT: The Muppets straight out of Law School},
  author={Chalkidis et al. (2020)},
  year={2020},
  note={arXiv:2010.02559}
}
```

- arXiv: 2010.02559