Casefacts Eval

casefacts-eval

CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval — Putta et al. (2026) (arXiv:2601.17230, 2026)

What this evaluates

Evaluates LLMs and retrieval models on verifying colloquial legal claims against U.S. Supreme Court precedents, measuring both verdict prediction accuracy and the quality of retrieved supporting case evidence.

Datasets

• CaseFacts — total 6294; splits: train (-1), test (-1); repo https://github.com/idirlab/supreme-court-dataset

Metrics

• Verdict Score (primary) — range: [0, 1]
  • Composite metric combining verdict accuracy and evidence score to measure joint evidence-weighted verdict performance.
• Evidence Score — range: [0, 1]
  • Case recall metric measuring the overlap between predicted citing cases and gold supporting cases.
• Verdict Accuracy — range: [0, 1]
  • Accuracy of predicting the correct verdict (Supported, Refuted, or Overruled).
• Recall@1 — range: [0, 1]
  • Standard recall@1 metric measuring whether the ground truth case appears in the top-1 retrieved result.
• Recall@5 — range: [0, 1]
  • Standard recall@5 metric measuring whether the ground truth case appears in the top-5 retrieved results.
• Recall@10 — range: [0, 1]
  • Standard recall@10 metric measuring whether the ground truth case appears in the top-10 retrieved results.

Input / output format

Input: A colloquial legal claim and a constrained list of 3,299 valid U.S. Supreme Court case names.

Output: A predicted verdict (Supported, Refuted, or Overruled), and a ranked list of citing Supreme Court cases from the provided list.

Scoring recipe

def compute_metrics(preds, golds):
    verdict_correct = [p['verdict'] == g['verdict'] for p, g in zip(preds, golds)]
    verdict_accuracy = sum(verdict_correct) / len(preds)
    
    evidence_overlap = [len(set(p['cases']) & set(g['cases'])) / len(g['cases']) for p, g in zip(preds, golds)]
    evidence_score = sum(evidence_overlap) / len(preds)
    
    verdict_score = verdict_accuracy * evidence_score
    
    recall_k = {}
    for k in [1, 5, 10]:
        hits = [1 if g['case_id'] in [c['id'] for c in p['cases'][:k]] else 0 for p, g in zip(preds, golds)]
        recall_k[f'Recall@{k}'] = sum(hits) / len(preds)
    return verdict_accuracy, evidence_score, verdict_score, recall_k

Common pitfalls

• Unrestricted web search degrades performance by retrieving noisy or non-authoritative cases outside the constrained 3,299-case gold list, lowering evidence overlap.
• LLMs frequently fail to output in the requested format when evidence is withheld, causing high error rates in naive factuality checks without retrieval.
• High verdict accuracy masks poor evidence retrieval, making the composite verdict score necessary for reliable fact-checking evaluation.

Evidence (verbatim from paper)

From Table[5], it is evident that the major challenge for this benchmark dataset is gathering evidence, as the evidence score (case recall metric) is much lower than the verdict accuracy. This points to the verdicts being easier to predict by the LLM, as both search baselines perform similarly on the verdict prediction. However, for a fact-checking application, the quality of the evidence retrieved is quite important for users’ trustworthiness, hence why we use the composite metric of “verdict score” as our primary metric for this dataset.

Citation

@misc{putta2026casefacts,
  title={CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval},
  author={Putta et al. (2026)},
  year={2026},
  note={arXiv:2601.17230}
}

• arXiv: 2601.17230