Nci Document Retrieval Eval

nci-document-retrieval-eval

A Neural Corpus Indexer for Document Retrieval — Wang et al. (2022) (arXiv:2206.02743, 2022)

What this evaluates

Evaluates a model's ability to retrieve relevant documents from a large corpus given a natural language query. It measures ranking quality and recall at various cutoffs to assess end-to-end document retrieval performance.

Datasets

• NQ320k — total 320000; splits: train (-1), val (-1)
• TriviaQA — total 78000; splits: train (-1), val (-1)

Metrics

• Recall@1 (primary) — range: [0, 1]
  • Measures how often the desired document is hit by the top-1 retrieved candidate.
• Recall@10 — range: [0, 1]
  • Measures how often the desired document is hit by the top-10 retrieved candidates.
• Recall@100 — range: [0, 1]
  • Measures how often the desired document is hit by the top-100 retrieved candidates.
• MRR@100 — range: [0, 1]
  • Calculates the reciprocal of the rank at which the first relevant document is retrieved, computed over the top-100 results.
• R-Precision — range: [0, 1]
  • Precision after R documents have been retrieved, where R is the number of relevant documents for the query.

Input / output format

Input: Natural language query.

Output: A ranked list of document identifiers (or documents) retrieved for the query.

Scoring recipe

def compute_metrics(predictions, gold):
    recalls, mrrs, r_precs = [], [], []
    for pred, rels in zip(predictions, gold):
        top100 = pred[:100]
        for N in [1, 10, 100]:
            recalls.append(len(set(pred[:N]) & set(rels)) / len(rels))
        mrr = 0.0
        for i, doc in enumerate(top100):
            if doc in rels:
                mrr = 1.0 / (i + 1)
                break
        mrrs.append(mrr)
        R = len(rels)
        topR = pred[:R]
        r_precs.append(len(set(topR) & set(rels)) / R)
    return {
        'Recall@1': sum(recalls[::3]) / len(recalls[::3]),
        'Recall@10': sum(recalls[1::3]) / len(recalls[1::3]),
        'Recall@100': sum(recalls[2::3]) / len(recalls[2::3]),
        'MRR@100': sum(mrrs) / len(mrrs),
        'R-Precision': sum(r_precs) / len(r_precs)
    }

Common pitfalls

• Recall is evaluated at different cutoffs (@1, @10, @100) depending on the dataset; ensure the correct cutoff is used for each benchmark.
• R-Precision uses R = number of relevant documents per query, which can be >1 in TriviaQA, unlike standard single-relevance benchmarks.
• MRR is computed over the top-100 results, not the full generated list.

Evidence (verbatim from paper)

Recall@N measures how often the desired document is hit by the top-N retrieved candidates. MRR calculates the reciprocal of the rank at which the first relevant document is retrieved. R-Precision is the precision after R documents have been retrieved, where R is the number of relevant documents for the query.

Citation

@misc{wang2022neuralcorpusindexer,
  title={A Neural Corpus Indexer for Document Retrieval},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2206.02743}
}

• arXiv: 2206.02743