# Legal RAG Bench Eval

> Evaluates end-to-end performance of legal retrieval-augmented generation systems by measuring retrieval accuracy, answer correctness, and answer groundedness. It uses a full factorial design across embedding and generative models, and introduces a hierarchical error decomposition to isolate hallucinations, retrieval failures, and reasoning errors. Use when the user wants to benchmark on Legal RAG Bench, or asks about evaluating this task. Reports correctness.

- Skill: `qhjqhj00/legal-rag-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/legal-rag-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/legal-rag-bench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/legal-rag-bench-eval

---


# legal-rag-bench-eval

> Legal RAG Bench: an end-to-end benchmark for legal RAG — Butler et al. (2026) (arXiv:2603.01710, 2026)

## What this evaluates

Evaluates end-to-end performance of legal retrieval-augmented generation systems by measuring retrieval accuracy, answer correctness, and answer groundedness. It uses a full factorial design across embedding and generative models, and introduces a hierarchical error decomposition to isolate hallucinations, retrieval failures, and reasoning errors.

## Datasets

- **Legal RAG Bench** — total 100; splits: test (100); repo https://github.com/isaacus-dev/legal-rag-bench

## Metrics

- `correctness` **(primary)** — range: [0, 1]
  - Binary metric: 1 if the generated answer entails the reference answer, 0 otherwise. Evaluated using an LLM-as-a-judge (GPT-5.2 high reasoning mode) against a clear rubric.
- `groundedness` — range: [0, 1]
  - Binary metric: 1 if the generated answer is supported by the retrieved passages provided to the model, 0 otherwise. Evaluated using an LLM-as-a-judge.
- `retrieval_accuracy` — range: [0, 1]
  - Binary metric: 1 if the annotated supporting passage is successfully retrieved by the embedding model, 0 otherwise.

## Input / output format

**Input**: A legal question and a set of retrieved text passages generated by an embedding model.

**Output**: A long-form natural language answer to the legal question.

## Scoring recipe

```python
def score_instance(question, retrieved_passages, generated_answer, reference_answer, annotated_passage):
    retrieval_acc = 1 if annotated_passage in retrieved_passages else 0
    groundedness = 1 if llm_judge(generated_answer, retrieved_passages) == 'supported' else 0
    correctness = 1 if llm_judge(generated_answer, reference_answer) == 'entails' else 0
    if not groundedness:
        error_type = 'Hallucination'
    elif not correctness and not retrieval_acc:
        error_type = 'Retrieval error'
    elif not correctness and retrieval_acc:
        error_type = 'Reasoning error'
    else:
        error_type = 'None'
    return correctness, groundedness, retrieval_acc, error_type
```

## Common pitfalls

- Overall RAG accuracy can mask hallucinations that coincidentally produce correct answers, making component-level metrics essential.
- Groundedness is evaluated strictly against the retrieved passages, not necessarily the ground-truth relevant passages, meaning a model can be marked grounded even if it retrieves irrelevant text.
- Retrieval accuracy only checks for the annotated supporting passage, ignoring the possibility that other retrieved passages might also be relevant.

## Evidence (verbatim from paper)

> In particular, for each question $i$, embedding model $e$, and LLM $l$, we assessed the following evaluation dimensions: 1. Correctness $(c_{eli})$: 1 if the model's answer entails the reference answer; 0 otherwise. 2. Groundedness $(g_{eli})$: 1 if the answer is supported by the retrieved passages provided to the model (irrespective of whether those passages are actually relevant); 0 otherwise. 3. Retrieval accuracy $(r_{ei})$: 1 if the annotated supporting passage is retrieved by the embedding model; 0 otherwise.

## Citation

```bibtex
@misc{butler2026legalragbench,
  title={Legal RAG Bench: an end-to-end benchmark for legal RAG},
  author={Butler et al. (2026)},
  year={2026},
  note={arXiv:2603.01710}
}
```

- arXiv: 2603.01710