Bbsard Eval

bbsard-eval

Bilingual BSARD: Extending Statutory Article Retrieval to Dutch — Lotfi et al. (2024) (arXiv:2412.07462, 2024)

What this evaluates

Evaluates the ability of retrieval models to match statutory article questions to the correct legal articles in Dutch and French. It benchmarks both zero-shot dense/lexical models and fine-tuned language-specific models on a parallel bilingual dataset.

Datasets

• bBSARD — total ?; splits: test (-1); repo https://github.com/nerses28/bBSARD

Metrics

• R@k, MAP@k, MRR@k, nDCG@k (primary) — range: percent
  • Standard retrieval metrics. R@k measures the fraction of queries where the relevant document appears in the top-k results. MAP@k averages precision at ranks where relevant documents appear. MRR@k averages the inverse of the rank of the first relevant document. nDCG@k measures ranking quality by discounting gains logarithmically based on position.

Input / output format

Input: A natural language question describing a legal scenario and a corpus of statutory articles (documents) to retrieve from.

Output: A ranked list of statutory articles corresponding to the query.

Scoring recipe

def compute_metrics(retrieved_docs, relevant_docs, k_values=[10, 100, 200, 500]):
    metrics = {}
    for k in k_values:
        top_k = retrieved_docs[:k]
        metrics[f'R@{k}'] = sum(1 for d in relevant_docs if d in top_k) / len(relevant_docs)
        # MAP, MRR, nDCG computed per query and macro-averaged across all queries
    return metrics

Common pitfalls

• Models with ≤512 token limits require chunking (200 tokens, 20 overlap), which can fragment context and hurt retrieval.
• Evaluation distinguishes between zero-shot and fine-tuned settings; mixing them invalidates comparisons.
• Metrics are macro-averaged across all queries, not micro-averaged or per-language.

Evidence (verbatim from paper)

To assess the performance of our models, we employ standard retrieval metrics: macro-averaged recall@k (R@k), mean average precision@k (MAP@k), mean reciprocal rank@k (MRR@k), and normalized discounted cumulative gain@k (nDCG@k).

Citation

@misc{lotfi2024bilingualbsard,
  title={Bilingual BSARD: Extending Statutory Article Retrieval to Dutch},
  author={Lotfi et al. (2024)},
  year={2024},
  note={arXiv:2412.07462}
}

• arXiv: 2412.07462