Legalbench RAG Eval

legalbench-rag-eval

LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain — Pipitone et al. (2024) (arXiv:2408.10343, 2024)

What this evaluates

Evaluates the retrieval fidelity of RAG systems in the legal domain by measuring how precisely and completely a model retrieves minimal, highly relevant text snippets from legal documents to answer specific queries.

Datasets

• LegalBench-RAG — total 6858; splits: test (-1); repo https://github.com/zeroentropy-cc/legalbenchrag

Metrics

• Precision (primary) — range: percent
  • Fraction of retrieved chunks in the top-k that are relevant to the query. Calculated as |relevant ∩ top_k| / k.
• Recall — range: percent
  • Fraction of all relevant chunks in the document that are successfully retrieved within the top-k results. Calculated as |relevant ∩ top_k| / |relevant|.

Input / output format

Input: A legal query paired with a legal document (or corpus) from which the system must retrieve relevant text chunks.

Output: A ranked list of top-k retrieved text chunks/snippets.

Scoring recipe

def score(retrieved, gold, k):
    top_k = retrieved[:k]
    hits = len(set(top_k) & set(gold))
    prec = (hits / k) * 100
    rec = (hits / len(gold)) * 100
    return prec, rec
# Average per dataset, then equally weight datasets

Common pitfalls

• Averaging scores globally across datasets instead of weighting each dataset equally regardless of size, as explicitly required.
• Failing to account for chunking strategy (fixed-size vs. recursive splitter) and reranker presence, which drastically shift Precision/Recall.
• Choosing k without balancing context richness against noise/hallucination risks, as noted in post-processing design decisions.

Evidence (verbatim from paper)

Performance comparison on different datasets for Precision and Recall at various k values for the Naive Method.

Citation

@misc{pipitone2024legalbenchrag,
  title={LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain},
  author={Pipitone et al. (2024)},
  year={2024},
  note={arXiv:2408.10343}
}

• arXiv: 2408.10343