Synlexlm Eval

synlexlm-eval

SynLexLM: Scaling Legal LLMs with Synthetic Data and Curriculum Learning — Upadhyay et al. (2025) (arXiv:2504.18762, 2025)

What this evaluates

Evaluates the impact of curriculum learning and synthetic data generation on legal LLM fine-tuning. It measures performance across legal summarization, classification, and question-answering benchmarks to determine if synthetic data improves model capabilities over real-data-only baselines.

Datasets

• EurLex-Sum — total ?; splits: train (-1), test (-1)
• EurLex — total ?; splits: train (-1), test (-1)
• LexGLUE — total ?; splits: test (-1)
• BigLaw-Bench — total ?; splits: test (-1)
• CUAD — total ?; splits: test (-1)

Metrics

• training loss (primary) — range: other
  • Average cross-entropy or MSE loss computed over 10 fine-tuning epochs, reported as the final value after convergence.

Input / output format

Input: Legal documents (EurLex/EurLex-Sum) for synthetic QA generation and fine-tuning; benchmark-specific prompts for classification, QA, and summarization tasks.

Output: Fine-tuned model predictions on target tasks; generated synthetic QA pairs for training.

Scoring recipe

# Training loss evaluation (Table 1)
total_loss = 0.0
num_batches = 0
for epoch in range(10):
    for batch in dataloader:
        preds = model(batch.input)
        loss = compute_loss(preds, batch.labels)
        total_loss += loss
        num_batches += 1
final_training_loss = total_loss / num_batches

Common pitfalls

• Only preliminary training loss is reported; final benchmark scores (e.g., accuracy, ROUGE) are pending and not yet published.
• Synthetic QA pairs are generated via an external LLM (Gemini Pro), which may introduce factual inaccuracies or biases into the training corpus.
• Curriculum learning complexity metrics (document length, concept density) are mentioned but not fully specified in the protocol.

Evidence (verbatim from paper)

Quantitative: Model performance on summarization tasks (on EurLex-Sum (Aumiller et al., [2022])) and classification and Q&A tasks (on LexGLUE (Chalkidis et al., [2022]), BigLaw-Bench (AI, [2024a])). We also plan to assess efficiency (training time, data usage). Table 1. Comparison of final training losses after 10 epochs between Baseline (only data from datasets) and SynLexLM (combination of synthetic data generation and curriculum learning).

Citation

@misc{upadhyay2025synlexlm,
  title={SynLexLM: Scaling Legal LLMs with Synthetic Data and Curriculum Learning},
  author={Upadhyay et al. (2025)},
  year={2025},
  note={arXiv:2504.18762}
}

• arXiv: 2504.18762