Greekbarbench Eval

greekbarbench-eval

GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations — Chlapanis et al. (2025) (arXiv:2505.17267, 2025)

What this evaluates

This benchmark probes free-text legal reasoning and multi-hop statutory citation using Greek Bar exam questions. It requires models to analyze case facts, cite relevant Greek legal articles, and produce open-ended legal analysis. Performance is measured across three dimensions: factual accuracy, correct statutory citation, and quality of legal reasoning.

Datasets

• GreekBarBench — total 310; splits: test (310); repo https://github.com/nlpaueb/greek-bar-bench

Metrics

• Facts — range: [1, 10]
  • 1–10 score based on a span-based rubric evaluating the accuracy of case facts mentioned in the response.
• Articles — range: [1, 10]
  • 1–10 score based on a span-based rubric evaluating the correct citation of Greek statutory articles.
• Analysis — range: [1, 10]
  • 1–10 score based on a span-based rubric evaluating the quality of legal reasoning and open-ended analysis.
• Mean (primary) — range: [1, 10]
  • Arithmetic average of the Facts, Articles, and Analysis scores.

Input / output format

Input: System prompt instructing citation of Greek statutory articles, followed by a user prompt containing numbered case facts, a specific question, and relevant legal context (statutory articles and distractors).

Output: Free-text response containing legal analysis, case fact references, and citations to Greek statutory articles, formatted according to the benchmark's prompt template.

Scoring recipe

def evaluate(response, gold_facts, gold_articles, gold_analysis):
    judge_prompt = build_span_judge_prompt(response, gold_facts, gold_articles, gold_analysis)
    judge_output = call_llm_judge(judge_prompt, model='GPT-4.1-mini')
    facts_score = parse_score(judge_output, 'Facts')
    articles_score = parse_score(judge_output, 'Articles')
    analysis_score = parse_score(judge_output, 'Analysis')
    mean_score = (facts_score + articles_score + analysis_score) / 3
    return {'Facts': facts_score, 'Articles': articles_score, 'Analysis': analysis_score, 'Mean': mean_score}

Common pitfalls

• LLM-judge scores are not directly comparable to human expert scores due to different dataset sizes (310 vs 22) and scoring methodologies.
• Model performance heavily depends on the legal context setting; evaluating in the 'no-context' setting tests parametric knowledge rather than reasoning with provided statutes, while 'oracle context' artificially inflates citation scores.
• The passing threshold is 6.00 across all dimensions, but models often fail specific dimensions (e.g., Articles/Analysis) even if the Mean passes.

Evidence (verbatim from paper)

We use GPT-4.1-mini as the judge with the Span-Judge prompt (§[3.2]), reporting the Facts score, Articles score, Analysis score and Mean score with standard error from three runs.

Citation

@misc{chlapanis2025greekbarbench,
  title={GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations},
  author={Chlapanis et al. (2025)},
  year={2025},
  note={arXiv:2505.17267}
}

• arXiv: 2505.17267