Dureader Retrieval Eval

dureader-retrieval-eval

DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine — Qiu et al. (2022) (arXiv:2203.10232, 2022)

What this evaluates

Passage retrieval for web search queries, evaluating a model's ability to rank relevant documents from a large collection. It probes in-domain retrieval accuracy as well as out-of-domain and cross-lingual generalization, highlighting challenges like salient phrase mismatch, syntactic mismatch, and false negatives.

Datasets

• DuReader_retrieval — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/baidu/DuReader

Metrics

• MRR@10 (primary) — range: [0, 1]
  • Mean Reciprocal Rank for the top 10 retrieved documents. Calculated as the average of 1/rank for the first relevant passage in the top-10 results across all queries.
• Recall@1 — range: [0, 1]
  • Recall for the top-1 retrieved item. The fraction of queries where the single most relevant passage is ranked first.
• Recall@50 — range: [0, 1]
  • Recall for the top-50 retrieved items. The fraction of queries where at least one relevant passage appears in the top-50 results.

Input / output format

Input: A query (Chinese or English) and a candidate passage (for re-ranking) or a passage collection (for retrieval).

Output: A ranked list of passages corresponding to the query.

Scoring recipe

def compute_metrics(predictions, gold_indices):
    mrr_scores = []
    recall_1_scores = []
    recall_50_scores = []
    for pred, gold in zip(predictions, gold_indices):
        if gold in pred[:10]:
            rank = pred.index(gold) + 1
            mrr_scores.append(1.0 / rank)
        else:
            mrr_scores.append(0.0)
        recall_1_scores.append(1.0 if pred[0] == gold else 0.0)
        recall_50_scores.append(1.0 if gold in pred[:50] else 0.0)
    return {
        'MRR@10': sum(mrr_scores) / len(mrr_scores),
        'Recall@1': sum(recall_1_scores) / len(recall_1_scores),
        'Recall@50': sum(recall_50_scores) / len(recall_50_scores)
    }

Common pitfalls

• The dataset contains false negatives (relevant passages incorrectly labeled as irrelevant), which can artificially suppress model scores if not accounted for.
• Dense retrievers trained on this dataset show poor zero-shot cross-domain generalization, often performing worse than BM25 without fine-tuning on the target domain.
• Cross-lingual retrieval (English query to Chinese passage) is significantly harder than monolingual retrieval due to lack of shared lexical terms, requiring semantic matching rather than simple term overlap.

Evidence (verbatim from paper)

We use the following evaluation metrics in our experiments: (1) Mean Reciprocal Rank for the top 10 retrieved documents (MRR@10), (2) Recall for the top-1 retrieved items (Recall@1) and (3) Recall for the top-50 retrieved items (Recall@50). Recall@50 is more suitable for evaluating the first-stage retrievers, while MRR@10 and Recall@1 are more suitable for assessing the second-stage re-rankers.

Citation

@misc{qiu2022dureader_retrieval,
  title={DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine},
  author={Qiu et al. (2022)},
  year={2022},
  note={arXiv:2203.10232}
}

• arXiv: 2203.10232