Legal Bert Eval

legal-bert-eval

LEGAL-BERT: The Muppets straight out of Law School — Chalkidis et al. (2020) (arXiv:2010.02559, 2020)

What this evaluates

Evaluates domain-adapted BERT models on legal text classification and named entity recognition to measure the impact of further pre-training and hyperparameter tuning strategies.

Datasets

• EURLEX57K — total ?; splits: train (-1), val (-1), test (-1)
• ECHR-CASES — total ?; splits: train (-1), val (-1), test (-1)
• CONTRACTS-NER — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy, F1 (primary) — range: [0, 1]
  • Standard accuracy and F1 scores for multi-label classification and sequence tagging, as replicated from prior work (Chalkidis et al., 2019).

Input / output format

Input: Raw legal text documents for classification tasks; tokenized text sequences for named entity recognition.

Output: Predicted class labels (binary or multi-label) for classification; predicted entity tags for sequence tagging.

Scoring recipe

# For classification tasks
acc = sum(pred == gold for pred, gold in zip(predictions, gold_labels)) / len(gold_labels)
# For NER tasks
f1 = compute_f1(predictions, gold_labels) # Standard BIO tag matching
# Aggregate across datasets/tasks as needed

Common pitfalls

• Blindly following default BERT hyperparameters (learning rate, epochs, batch size) without domain-specific tuning.
• Using a fixed maximum number of training epochs (e.g., 4) which can cause underfitting on legal datasets; early stopping on validation loss is recommended.

Evidence (verbatim from paper)

We evaluate our models on text classification and sequence tagging using three datasets. EURLEX57K (Chalkidis et al., 2019b) is a large-scale multi-label text classification dataset of EU laws, also suitable for few and zero-shot learning. ECHR-CASES (Chalkidis et al., 2019a) contains cases from the European Court of Human Rights (Aletras et al., 2016) and can be used for binary and multi-label text classification. Finally, CONTRACTS-NER (Chalkidis et al., 2017, 2019d) is a dataset for named entity recognition on US contracts consisting of three subsets, contract header, dispute resolution, and lease details. We replicate the experiments of Chalkidis et al. (2019c,a,d) when fine-tuning BERT for all datasets.

Citation

@misc{chalkidis2020legalbert,
  title={LEGAL-BERT: The Muppets straight out of Law School},
  author={Chalkidis et al. (2020)},
  year={2020},
  note={arXiv:2010.02559}
}

• arXiv: 2010.02559