Turkish Legal Retrieval Eval

turkish-legal-retrieval-eval

Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain — Ozgür Uğur et al. (2026) (arXiv:2601.16018, 2026)

What this evaluates

Evaluates the quality of Turkish legal language model embeddings for information retrieval tasks, specifically focusing on case law, regulations, and contract retrieval. It also assesses masked language modeling capabilities on diverse Turkish corpora to measure morphological and domain-specific token prediction accuracy.

Datasets

• MTEB-Turkish benchmark — total ?; splits: test (-1)
• Turkish Legal Retrieval Benchmarks — total ?; splits: test (-1)
• blackerx/turkish_v2 — total ?; splits: test (-1); HF blackerx/turkish_v2
• fthbrmnby/turkish_product_reviews — total ?; splits: test (-1); HF fthbrmnby/turkish_product_reviews
• hazal/Turkish-Biomedical-corpus-trM — total ?; splits: test (-1); HF hazal/Turkish-Biomedical-corpus-trM
• newmindai/EuroHPC-Legal — total ?; splits: test (-1); HF newmindai/EuroHPC-Legal

Metrics

• MTEB Score (primary) — range: percent
  • Mean TaskType score calculated as the average across Classification, Clustering, Pair Classification, Retrieval, and STS tasks.
• Legal Score — range: percent
  • Retrieval performance specifically evaluated on legal domain documents (case law, regulations, contracts).
• Contracts — range: percent
  • Retrieval performance specifically on contract document subsets.
• MLM Accuracy — range: percent
  • Percentage of correctly predicted masked tokens using a 15% masking rate with an 80-10-10 replacement strategy.
• Perplexity (PP) — range: other
  • Standard perplexity metric measuring the model's prediction uncertainty over the test corpus.

Input / output format

Input: For retrieval tasks: query and document pairs or single documents for embedding. For MLM: Turkish text sequences with 15% of token positions randomly masked.

Output: For retrieval: dense embedding vectors or ranked document lists. For MLM: predicted token IDs for masked positions.

Scoring recipe

def compute_metrics(predictions, gold):
    # MTEB Score: Mean across task types
    mteb_score = mean([score(task) for task in ['Classification', 'Clustering', 'PairClassification', 'Retrieval', 'STS']])
    # Legal & Contracts: Domain-specific retrieval scores
    legal_score = score('Retrieval', domain='legal')
    contracts_score = score('Retrieval', domain='contracts')
    # MLM Accuracy: Token prediction accuracy
    mlm_acc = (sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)) * 100
    return mteb_score, legal_score, contracts_score, mlm_acc

Common pitfalls

• Optimizing for MLM loss does not guarantee better downstream retrieval performance; intermediate checkpoints often outperform those with minimal pre-training loss.
• Sequence length must be matched to downstream task requirements (e.g., 2048 for legal retrieval) rather than training data distribution, as shorter lengths cause significant performance drops in legal domains.
• Evaluation datasets for MLM must match cased/uncased configurations; mixing them leads to artificially low scores.

Evidence (verbatim from paper)

Table [12] presents the performance on Turkish retrieval benchmarks across these versions, where MTEB Score is calculated as Mean TaskType.

Citation

@misc{ugur2026mecellem,
  title={Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain},
  author={Ozgür Uğur et al. (2026)},
  year={2026},
  note={arXiv:2601.16018}
}

• arXiv: 2601.16018