qhjqhj00 japanese-bar-exam-legal-reasoning-eval
Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task — Choi et al. (2026) (arXiv:2604.23730, 2026)
What this evaluates
Evaluates open-ended legal reasoning capabilities of LLMs in the Japanese legal domain. It assesses their ability to generate structured, legally accurate arguments based on bar exam writing tasks.
Datasets
- Japanese Bar Exam Writing Task — total 21; splits: test (21)
Metrics
expert_score(primary) — range: [0, 100]- Holistic numerical score from 0 to 100 assigned by subject-specific legal experts, assessing issue identification, legal analysis, application of law to facts, logical structure, and clarity.
pass_rate— range: percent- Percentage of model outputs scoring at or above the 25-point threshold, which corresponds to the minimum passing requirement in the real Japanese bar exam.
Input / output format
Input: A bar exam question from 2021-2023, presented without modification, preceded by an instruction specifying the temporal scope of applicable law. A system prompt assigns the role of a Japanese legal expert and requires outputs based on Japanese law in casual form.
Output: Open-ended legal reasoning text written in casual Japanese, structured to address issue identification, legal analysis, application to facts, and logical argumentation.
Scoring recipe
def evaluate(answer, question, subject):
expert = assign_expert(subject) # 7 experts for 7 subjects
score = expert.holistic_grade(
answer,
criteria=["issue_identification", "legal_analysis",
"application_to_facts", "logical_structure", "clarity"]
)
if score >= 75: grade = "Excellent"
elif score >= 58: grade = "Good"
elif score >= 42: grade = "Adequate"
else: grade = "Poor"
return {"score": score, "grade": grade, "pass": score >= 25}
Common pitfalls
- Experts are domain-specific (7 evaluators for 7 subjects), so cross-subject generalization of scores is invalid.
- The 25-point pass threshold is a hard cutoff for 'passing' but does not reflect the full distribution of expert scores.
- Temporal constraint instruction is critical; models must reason based on laws in force as of the exam year, not current law.
Evidence (verbatim from paper)
The outputs were assessed using both categorical gradings and numerical scores. The categorical gradings consist of four categories that generally correspond to the predefined score ranges in the bar exam grading policy: Excellent (100–75), Good (74–58), Adequate (57–42), and Poor (41–0). We treat 25 as an important threshold: in the real exam, a score below 25% is an instant failure, so an answer scoring at least 25 meets the minimum requirement.
Citation
@misc{choi2026expert,
title={Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task},
author={Choi et al. (2026)},
year={2026},
note={arXiv:2604.23730}
}
- arXiv: 2604.23730