Dpr Retrieval Eval

dpr-retrieval-eval

Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al. (2020) (arXiv:2004.04906, 2020)

What this evaluates

Evaluates a model's ability to retrieve relevant passages from a large unstructured corpus for open-domain question answering. It probes semantic matching and dense retrieval capabilities by measuring how often the correct answer span appears in the top-k retrieved passages.

Datasets

• Natural Questions — total ?; splits: train (58880), dev (8757), test (3610)
• TriviaQA — total ?; splits: train (60413), dev (8837), test (11313)
• WebQuestions — total ?; splits: train (2474), dev (361), test (2032)
• CuratedTREC — total ?; splits: train (1125), dev (133), test (694)
• SQuAD v1.1 — total ?; splits: train (70096), dev (8886), test (10570)

Metrics

• top-k retrieval accuracy (primary) — range: percent
  • Percentage of top-k retrieved passages that contain the gold answer span. Calculated as the number of questions where the answer is found in the top-k results divided by the total number of questions.

Input / output format

Input: A natural language question and a candidate pool of Wikipedia passages (100-word blocks, each prepended with the article title and a [SEP] token).

Output: A ranked list of candidate passages for the given question.

Scoring recipe

def compute_topk_accuracy(predictions, gold_answers, k=20):
    correct = 0
    for pred_passages, gold_span in zip(predictions, gold_answers):
        top_k = pred_passages[:k]
        if any(gold_span in passage for passage in top_k):
            correct += 1
    return correct / len(gold_answers)

Common pitfalls

• Wikipedia version and preprocessing differences require filtering out questions where gold passages cannot be matched to the candidate pool.
• For TREC, WebQuestions, and TriviaQA, positive passages are selected using BM25 top-100 results containing the answer, introducing a retrieval bias.
• SQuAD is explicitly noted as suboptimal for open-domain QA because many questions lack sufficient context without the original provided paragraph.

Evidence (verbatim from paper)

Table 2: Top-20 & Top-100 retrieval accuracy on test sets, measured as the percentage of top 20/100 retrieved passages that contain the answer.

Citation

@misc{karpukhin2020dpr,
  title={Dense Passage Retrieval for Open-Domain Question Answering},
  author={Karpukhin et al. (2020)},
  year={2020},
  note={arXiv:2004.04906}
}

• arXiv: 2004.04906