Legal RAG Bench Eval

legal-rag-bench-eval

Legal RAG Bench: an end-to-end benchmark for legal RAG — Butler et al. (2026) (arXiv:2603.01710, 2026)

What this evaluates

Evaluates end-to-end performance of legal retrieval-augmented generation systems by measuring retrieval accuracy, answer correctness, and answer groundedness. It uses a full factorial design across embedding and generative models, and introduces a hierarchical error decomposition to isolate hallucinations, retrieval failures, and reasoning errors.

Datasets

• Legal RAG Bench — total 100; splits: test (100); repo https://github.com/isaacus-dev/legal-rag-bench

Metrics

• correctness (primary) — range: [0, 1]
  • Binary metric: 1 if the generated answer entails the reference answer, 0 otherwise. Evaluated using an LLM-as-a-judge (GPT-5.2 high reasoning mode) against a clear rubric.
• groundedness — range: [0, 1]
  • Binary metric: 1 if the generated answer is supported by the retrieved passages provided to the model, 0 otherwise. Evaluated using an LLM-as-a-judge.
• retrieval_accuracy — range: [0, 1]
  • Binary metric: 1 if the annotated supporting passage is successfully retrieved by the embedding model, 0 otherwise.

Input / output format

Input: A legal question and a set of retrieved text passages generated by an embedding model.

Output: A long-form natural language answer to the legal question.

Scoring recipe

def score_instance(question, retrieved_passages, generated_answer, reference_answer, annotated_passage):
    retrieval_acc = 1 if annotated_passage in retrieved_passages else 0
    groundedness = 1 if llm_judge(generated_answer, retrieved_passages) == 'supported' else 0
    correctness = 1 if llm_judge(generated_answer, reference_answer) == 'entails' else 0
    if not groundedness:
        error_type = 'Hallucination'
    elif not correctness and not retrieval_acc:
        error_type = 'Retrieval error'
    elif not correctness and retrieval_acc:
        error_type = 'Reasoning error'
    else:
        error_type = 'None'
    return correctness, groundedness, retrieval_acc, error_type

Common pitfalls

• Overall RAG accuracy can mask hallucinations that coincidentally produce correct answers, making component-level metrics essential.
• Groundedness is evaluated strictly against the retrieved passages, not necessarily the ground-truth relevant passages, meaning a model can be marked grounded even if it retrieves irrelevant text.
• Retrieval accuracy only checks for the annotated supporting passage, ignoring the possibility that other retrieved passages might also be relevant.

Evidence (verbatim from paper)

In particular, for each question $i$, embedding model $e$, and LLM $l$, we assessed the following evaluation dimensions: 1. Correctness $(c_{eli})$: 1 if the model's answer entails the reference answer; 0 otherwise. 2. Groundedness $(g_{eli})$: 1 if the answer is supported by the retrieved passages provided to the model (irrespective of whether those passages are actually relevant); 0 otherwise. 3. Retrieval accuracy $(r_{ei})$: 1 if the annotated supporting passage is retrieved by the embedding model; 0 otherwise.

Citation

@misc{butler2026legalragbench,
  title={Legal RAG Bench: an end-to-end benchmark for legal RAG},
  author={Butler et al. (2026)},
  year={2026},
  note={arXiv:2603.01710}
}

• arXiv: 2603.01710