qhjqhj00 legal-cloze-test-eval
HUKUKBERT: Domain-Specific Language Model for Turkish Law — Mehmet Utku ÖZTÜRK et al. (2026) (arXiv:2604.04790, 2026)
What this evaluates
This benchmark probes a model's ability to understand and predict precise legal terminology and procedural concepts in Turkish court documents. It evaluates both masked language modeling capabilities on legal cloze sentences and structural segmentation accuracy for parsing document sections.
Datasets
- Legal Cloze Test benchmark — total 750; splits: test (750)
- v12 Court Decision Segmentation Dataset — total ?; splits: test (-1)
Metrics
Top-1 Accuracy(primary) — range: percent- Percentage of instances where the model's highest-probability prediction exactly matches the ground-truth masked token.
Document Pass Rate (doc_pass)— range: percent- Strict document-level binary metric: 1 if all structural boundary predictions in a document are correct, 0 otherwise. Averaged across documents.
Input / output format
Input: For cloze test: Turkish legal sentences with a single [MASK] token replacing a key legal term. For segmentation: full Turkish court decision documents tokenized for sequence labeling.
Output: For cloze test: a single predicted token. For segmentation: a sequence of structural labels (e.g., B-header, B-reasoning, B-ruling) for each token.
Scoring recipe
def score_cloze(predictions, golds):
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
return correct / len(golds)
def score_doc_pass(predictions, golds):
doc_passes = []
for pred_seq, gold_seq in zip(predictions, golds):
doc_passes.append(1 if all(p == g for p, g in zip(pred_seq, gold_seq)) else 0)
return sum(doc_passes) / len(doc_passes)
Common pitfalls
- Using minimum validation cross-entropy loss to select the best checkpoint, which peaks much earlier (epoch 0.64) than task-specific metrics like doc_pass (epoch 3.67).
- Confusing per-token boundary F1 with strict document-level pass rate; doc_pass requires every single boundary in a document to be correct, making it much stricter than token-level metrics.
- Assuming general-domain models scale linearly with data size; the benchmark shows they fail on legal semantic shifts regardless of pre-training token count.
Evidence (verbatim from paper)
The primary evaluation metric is the Document Pass Rate (doc_pass), a strict document-level binary metric where a single incorrect boundary prediction causes the entire document to fail.
Citation
@misc{ozturk2026hukukbert,
title={HUKUKBERT: Domain-Specific Language Model for Turkish Law},
author={Mehmet Utku ÖZTÜRK et al. (2026)},
year={2026},
note={arXiv:2604.04790}
}
- arXiv: 2604.04790