Public Defense Retrieval Eval

public-defense-retrieval-eval

Legal Retrieval for Public Defenders — Stammbach et al. (2026) (arXiv:2601.14348, 2026)

What this evaluates

Evaluates the ability of retrieval and reranking models to surface relevant appellate brief paragraphs for public defender search queries. It probes domain-specific adaptation, query expansion strategies, and the impact of synthetic data generation on legal information retrieval.

Datasets

• PD Dataset — total 170; splits: test (170); repo https://github.com/dominiksinsaarland/PublicDefenderRetrieval
• NJ OPD Dataset — total ?; splits: test (-1)
• BarExam-QA — total ?; splits: train (-1)
• LePaRD — total ?; splits: train (-1)

Metrics

• recall@5 (primary) — range: percent
  • Fraction of queries where at least one relevant passage appears in the top-5 retrieved results. Official metric for the PD dataset.
• recall@1 — range: percent
  • Fraction of queries where at least one relevant passage appears in the top-1 retrieved results.
• F1 score — range: percent
  • Harmonic mean of precision and recall for binary relevance judgment in reranking experiments.

Input / output format

Input: A natural language search query from a public defender, paired against a corpus of independent paragraphs from appellate briefs.

Output: For retrieval: a ranked list of top-k (k=1,5) candidate paragraphs. For reranking: a binary relevance label (relevant/irrelevant) for each candidate paragraph.

Scoring recipe

def recall_at_k(predictions, gold, k):
    hits = 0
    for q_preds, q_gold in zip(predictions, gold):
        if any(g in q_preds[:k] for g in q_gold):
            hits += 1
    return hits / len(predictions)

def f1_score(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Data leakage in synthetic data generation: failing to remove paragraphs already annotated as retrieval targets inflates performance.
• Domain shift: fine-tuning on general legal benchmarks (BarExam-QA, LePaRD) decreases performance on public defense queries.
• Reranker baseline: most off-the-shelf rerankers underperform a simple majority baseline because the dataset is heavily skewed toward relevant paragraphs.

Evidence (verbatim from paper)

Given the practitioner-facing nature of the PD dataset, we report recall@5 as the official metric for the PD dataset (the BriefBank by default also returns five search results).

Citation

@misc{stammbach2026legalretrieval,
  title={Legal Retrieval for Public Defenders},
  author={Stammbach et al. (2026)},
  year={2026},
  note={arXiv:2601.14348}
}

• arXiv: 2601.14348