Lexam Eval

lexam-eval

LEXam: Benchmarking Legal Reasoning on 340 Law Exams — Fan et al. (2025) (arXiv:2505.12864, 2025)

What this evaluates

Probes large language models' ability to perform structured, multi-step legal reasoning on real-world law exam questions. It evaluates both open-ended legal analysis and multiple-choice selection across diverse jurisdictions and legal domains.

Datasets

• LEXam — total 4886; splits: test (4886); repo https://github.com/LEXam-Benchmark/LEXam

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly selected answer labels out of total multiple-choice questions. Label distribution is balanced via permutation to ensure unbiased performance.
• LLM-as-a-Judge score — range: other
  • Minimum score assigned by an ensemble of GPT-4o, Qwen3-32B, and DeepSeek-V3 acting as judges. The judging prompt is expert-verified and calibrated to penalize plausible but incorrect legal information.

Input / output format

Input: Open questions: course name/knowledge domain, standard legal reasoning guidance, and the exam question. MCQs: course title, standard legal reasoning steps guidance, and the question with four permuted answer choices.

Output: Open questions: free-text structured legal reasoning response. MCQs: single selected answer label.

Scoring recipe

# MCQ Evaluation
mcq_accuracy = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

# Open Question Evaluation
def evaluate_open(model_output, question, domain):
    judge_prompt = build_judge_prompt(question, model_output, domain)
    scores = [llm_judge(judge_prompt) for llm_judge in [GPT4o, Qwen3_32B, DeepSeek_V3]]
    return min(scores)

Common pitfalls

• Using lexical/semantic metrics (BLEU, ROUGE, BERTScore) for open questions, which fail to capture correct legal reasoning due to inherently low lexical similarity.
• Relying on a single LLM judge instead of the required minimum-score ensemble, which undermines reliability and fails the Alternative Annotator Test validation.
• Ignoring answer choice permutation in MCQs, which is necessary to balance label distribution and ensure accuracy reflects true model capability rather than positional bias.

Evidence (verbatim from paper)

We use accuracy scores for the MCQ evaluations. Since the choice label distribution is balanced through permutation, accuracy reflects unbiased performance of LLMs. ... In our evaluation, we adopt an minimum-score ensemble of GPT-4o, Qwen3-32B, and DeepSeek-V3 to grade open questions.

Citation

@misc{fan2025lexam,
  title={LEXam: Benchmarking Legal Reasoning on 340 Law Exams},
  author={Fan et al. (2025)},
  year={2025},
  note={arXiv:2505.12864}
}

• arXiv: 2505.12864