# Bsard Eval

> Evaluates the ability of information retrieval models to rank relevant Belgian statutory legal articles in response to natural language citizen questions. It probes domain-specific legal retrieval, handling the challenge of mapping unstructured queries to structured hierarchical legal texts. Use when the user wants to benchmark on BSARD, or asks about evaluating this task. Reports Recall@100.

- Skill: `qhjqhj00/bsard-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/bsard-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/bsard-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/bsard-eval

---


# bsard-eval

> A Statutory Article Retrieval Dataset in French — Antoine Louis, Gerasimos Spanakis (2021) (arXiv:2108.11792, 2021)

## What this evaluates

Evaluates the ability of information retrieval models to rank relevant Belgian statutory legal articles in response to natural language citizen questions. It probes domain-specific legal retrieval, handling the challenge of mapping unstructured queries to structured hierarchical legal texts.

## Datasets

- **BSARD** — total 1100; splits: train (-1), test (-1); repo https://github.com/maastrichtlawtech/bsard

## Metrics

- `Recall@100` **(primary)** — range: percent
  - Recall@k measures the fraction of all relevant documents that appear in the top-k retrieved results. It is calculated as |relevant ∩ top_k| / |relevant|. Macro-averaged across queries.
- `Mean Average Precision@100` — range: percent
  - MAP@k computes the average precision across all queries. For a single query, precision is calculated at each rank where a relevant document appears, then averaged over all relevant documents in the top-k.
- `Mean Reciprocal Rank@100` — range: percent
  - MRR@k is the reciprocal of the rank of the first relevant document in the top-k results. If no relevant document is found in top-k, the score is 0.

## Input / output format

**Input**: Natural language question (query) and a candidate corpus of Belgian statutory articles.

**Output**: Ranked list of statutory articles (or top-k articles) returned by the retrieval model.

## Scoring recipe

```python
def compute_metrics(relevant_docs, ranked_docs, k=100):
    rel_set = set(relevant_docs)
    top_k = ranked_docs[:k]
    r_at_k = len(rel_set.intersection(top_k)) / len(rel_set) if rel_set else 0
    rr = 0.0
    for i, doc in enumerate(top_k):
        if doc in rel_set:
            rr = 1.0 / (i + 1)
            break
    ap = 0.0
    hits = 0
    for i, doc in enumerate(top_k):
        if doc in rel_set:
            hits += 1
            ap += hits / (i + 1)
    ap = ap / len(rel_set) if rel_set else 0
    return r_at_k, ap, rr
```

## Common pitfalls

- Questions have a variable number of relevant articles, making Precision@k at a fixed k misleading or always <1 if k < r.
- k must be chosen large enough (e.g., 100+) to adequately capture recall, as small k values artificially deflate performance.
- Pre-trained embeddings (e.g., CamemBERT) perform poorly without task-specific fine-tuning for this domain.

## Evidence (verbatim from paper)

> We use three standard information retrieval metrics (Manning et al., 2008) to evaluate performance, namely the (macro-averaged) recall@$k$ (R@$k$), mean average precision@$k$ (MAP@$k$), and mean reciprocal rank@$k$ (MRR@$k$). We deliberately omit to report the precision@$k$ given that questions have a variable number of relevant articles (see Figure 2(c)), which makes it senseless to report it at a fixed $k$ – questions with $r$ relevant articles will always have P@$k<1$ if $k>r$. For the same reason, $k$ should be large enough for the recall@$k$. Hence, we use $k\in{100,200,500}$ for our evaluation.

## Citation

```bibtex
@misc{louis2021bsard,
  title={A Statutory Article Retrieval Dataset in French},
  author={Antoine Louis, Gerasimos Spanakis (2021)},
  year={2021},
  note={arXiv:2108.11792}
}
```

- arXiv: 2108.11792