Housing Statute QA Eval

housing-statute-qa-eval

A Reasoning-Focused Legal Retrieval Benchmark — Zheng et al. (2025) (arXiv:2505.03970, 2025)

What this evaluates

Evaluates retrieval and reasoning over housing statutes, requiring models to connect queries to lexically distant legal texts and answer standardized Yes/No or categorical questions.

Datasets

• Housing Statute QA — total ?; splits: test (-1)

Metrics

• Recall@10 (primary) — range: [0, 1]
  • Measures whether the gold passage appears in the top 10 retrieved documents. Reported as a lower/upper bound range due to multiple valid gold passages per query.
• TF-IDF cosine similarity — range: [0, 1]
  • Computes the cosine similarity between TF-IDF vector representations of text pairs. Used to measure lexical overlap between (query, gold passage) and (gold passage, answer).

Input / output format

Input: A legal query or question paired with a candidate corpus of housing statutes or legal passages.

Output: A ranked list of retrieved passages (top-10) and/or a generated answer to the legal question.

Scoring recipe

def score_retrieval(predictions, gold):
    return 1.0 if gold in predictions else 0.0

def score_lexical(query, passage, answer=None):
    sim = tfidf_cosine_similarity(query, passage)
    if answer:
        sim_qa = tfidf_cosine_similarity(passage, answer)
        return sim, sim_qa
    return sim

Common pitfalls

• Recall is reported as a range (lower/upper bound) because multiple passages may be considered gold for a single query.
• Categorical answers are transformed to Yes/No for downstream evaluation, which affects lexical similarity calculations.
• Low lexical overlap requires semantic/reasoning-based retrieval rather than keyword matching.

Evidence (verbatim from paper)

We report the recall lower/upper bound for Housing Statute QA, see Section [5.2] for details. Table 3. Baseline retrieval performance (Recall@10) of BM25 (lexical) and E5-large-v2 (dense) retrieval methods on Bar Exam QA (aggregate), Housing Statute QA, NQ, HotpotQA, COLIEE, and CLERC. We use TF-IDF cosine similarity as the lexical similarity metric because it is a closely related metric to BM25, a strong lexical baseline ranking function for retrieval.

Citation

@misc{zheng2025reasoning,
  title={A Reasoning-Focused Legal Retrieval Benchmark},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2505.03970}
}

• arXiv: 2505.03970