qhjqhj00 bar-exam-qa-eval
A Reasoning-Focused Legal Retrieval Benchmark — Zheng et al. (2025) (arXiv:2505.03970, 2025)
What this evaluates
Evaluates a model's ability to retrieve relevant legal passages and answer legal questions that require multi-hop or analogical reasoning, characterized by low lexical overlap between queries and documents.
Datasets
- Bar Exam QA — total ?; splits: test (-1)
Metrics
Recall@10(primary) — range: [0, 1]- Measures whether the gold passage appears in the top 10 retrieved documents. Calculated as the fraction of queries where the gold passage is ranked within the top 10 results.
TF-IDF cosine similarity— range: [0, 1]- Computes the cosine similarity between TF-IDF vector representations of text pairs. Used to measure lexical overlap between (query, gold passage) and (gold passage, answer).
Input / output format
Input: A legal query or question paired with a candidate corpus of legal statutes or case passages.
Output: A ranked list of retrieved passages (top-10) and/or a generated answer to the legal question.
Scoring recipe
def score_retrieval(predictions, gold):
return 1.0 if gold in predictions else 0.0
def score_lexical(query, passage, answer=None):
sim = tfidf_cosine_similarity(query, passage)
if answer:
sim_qa = tfidf_cosine_similarity(passage, answer)
return sim, sim_qa
return sim
Common pitfalls
- Low lexical overlap means standard keyword-based retrievers (BM25) perform poorly; models must rely on semantic or reasoning capabilities.
- Housing Statute QA uses lower/upper bounds for recall because multiple passages may be considered gold for a single query.
- Downstream QA answers are transformed to Yes/No to standardize evaluation, which artificially inflates lexical similarity for categorical questions.
Evidence (verbatim from paper)
We report the recall lower/upper bound for Housing Statute QA, see Section [5.2] for details. Table 3. Baseline retrieval performance (Recall@10) of BM25 (lexical) and E5-large-v2 (dense) retrieval methods on Bar Exam QA (aggregate), Housing Statute QA, NQ, HotpotQA, COLIEE, and CLERC. We use TF-IDF cosine similarity as the lexical similarity metric because it is a closely related metric to BM25, a strong lexical baseline ranking function for retrieval.
Citation
@misc{zheng2025reasoning,
title={A Reasoning-Focused Legal Retrieval Benchmark},
author={Zheng et al. (2025)},
year={2025},
note={arXiv:2505.03970}
}
- arXiv: 2505.03970