Laiw Eval

laiw-eval

LAiW: A Chinese Legal Large Language Models Benchmark — Dai et al. (2023) (arXiv:2310.05620, 2023)

What this evaluates

Evaluates Chinese legal LLM capabilities across three hierarchical levels: basic legal NLP, basic legal application, and complex legal application. Probes tasks including named entity recognition, judicial summarization, case recognition, judgment prediction, legal question answering, and legal reasoning generation to measure domain-specific text processing, analysis, and reasoning skills.

Datasets

• LAiW Legal Evaluation Dataset (LED) — total 4564; splits: test (4564); repo https://github.com/Dai-shen/LAiW

Metrics

• Accuracy (primary) — range: [0, 1]
  • Task-specific standard metrics applied per task type: Accuracy or F1 for classification tasks (NER, CR, CJP, CTP, LQA), ROUGE or BLEU for generation tasks (JS, JRG, LC), and similarity scores for matching tasks (SCM, CFM).

Input / output format

Input: One-shot prompts containing legal documents, case descriptions, or questions, formatted according to the specific task type (e.g., NER, summarization, classification, generation).

Output: Task-specific model outputs: extracted legal entities/elements, summarized text, case type labels, recommended legal articles, judgment predictions, or legal reasoning/consultation answers.

Scoring recipe

def score(predictions, gold, task_type):
    if task_type in ['NER', 'CR', 'CJP', 'CTP', 'LQA']:
        return accuracy_or_f1(predictions, gold)
    elif task_type in ['JS', 'JRG', 'LC']:
        return rouge_or_bleu(predictions, gold)
    elif task_type in ['SCM', 'CFM']:
        return similarity_score(predictions, gold)
    else:
        raise ValueError('Unknown task type')

Common pitfalls

• Prompts are strictly one-shot; performance may vary significantly with zero-shot or different few-shot configurations.
• The benchmark uses a staged release; only the first stage (Basic Legal NLP) has publicly released LED sizes, while subsequent tasks lack explicit evaluation data sizes.
• Tasks span multiple legal domains but primarily focus on criminal and civil law, which may limit generalization to other legal fields.

Evidence (verbatim from paper)

To evaluate the aforementioned capabilities and their corresponding tasks, we integrate and construct the Legal Evaluation Dataset (LED) based on existing open-source data. ... Currently, all the tasks are in one-shot form, and people can see the examples in https://github.com/Dai-shen/LAiW.

Citation

@misc{dai2023laiw,
  title={LAiW: A Chinese Legal Large Language Models Benchmark},
  author={Dai et al. (2023)},
  year={2023},
  note={arXiv:2310.05620}
}

• arXiv: 2310.05620