Bsard Eval

bsard-eval

A Statutory Article Retrieval Dataset in French — Antoine Louis, Gerasimos Spanakis (2021) (arXiv:2108.11792, 2021)

What this evaluates

Evaluates the ability of information retrieval models to rank relevant Belgian statutory legal articles in response to natural language citizen questions. It probes domain-specific legal retrieval, handling the challenge of mapping unstructured queries to structured hierarchical legal texts.

Datasets

• BSARD — total 1100; splits: train (-1), test (-1); repo https://github.com/maastrichtlawtech/bsard

Metrics

• Recall@100 (primary) — range: percent
  • Recall@k measures the fraction of all relevant documents that appear in the top-k retrieved results. It is calculated as |relevant ∩ top_k| / |relevant|. Macro-averaged across queries.
• Mean Average Precision@100 — range: percent
  • MAP@k computes the average precision across all queries. For a single query, precision is calculated at each rank where a relevant document appears, then averaged over all relevant documents in the top-k.
• Mean Reciprocal Rank@100 — range: percent
  • MRR@k is the reciprocal of the rank of the first relevant document in the top-k results. If no relevant document is found in top-k, the score is 0.

Input / output format

Input: Natural language question (query) and a candidate corpus of Belgian statutory articles.

Output: Ranked list of statutory articles (or top-k articles) returned by the retrieval model.

Scoring recipe

def compute_metrics(relevant_docs, ranked_docs, k=100):
    rel_set = set(relevant_docs)
    top_k = ranked_docs[:k]
    r_at_k = len(rel_set.intersection(top_k)) / len(rel_set) if rel_set else 0
    rr = 0.0
    for i, doc in enumerate(top_k):
        if doc in rel_set:
            rr = 1.0 / (i + 1)
            break
    ap = 0.0
    hits = 0
    for i, doc in enumerate(top_k):
        if doc in rel_set:
            hits += 1
            ap += hits / (i + 1)
    ap = ap / len(rel_set) if rel_set else 0
    return r_at_k, ap, rr

Common pitfalls

• Questions have a variable number of relevant articles, making Precision@k at a fixed k misleading or always <1 if k < r.
• k must be chosen large enough (e.g., 100+) to adequately capture recall, as small k values artificially deflate performance.
• Pre-trained embeddings (e.g., CamemBERT) perform poorly without task-specific fine-tuning for this domain.

Evidence (verbatim from paper)

We use three standard information retrieval metrics (Manning et al., 2008) to evaluate performance, namely the (macro-averaged) recall@$k$ (R@$k$), mean average precision@$k$ (MAP@$k$), and mean reciprocal rank@$k$ (MRR@$k$). We deliberately omit to report the precision@$k$ given that questions have a variable number of relevant articles (see Figure 2(c)), which makes it senseless to report it at a fixed $k$ – questions with $r$ relevant articles will always have P@$k<1$ if $k>r$. For the same reason, $k$ should be large enough for the recall@$k$. Hence, we use $k\in{100,200,500}$ for our evaluation.

Citation

@misc{louis2021bsard,
  title={A Statutory Article Retrieval Dataset in French},
  author={Antoine Louis, Gerasimos Spanakis (2021)},
  year={2021},
  note={arXiv:2108.11792}
}

• arXiv: 2108.11792