Promptriever Retrieval Eval

promptriever-retrieval-eval

Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models — Weller et al. (2024) (arXiv:2409.11136, 2024)

What this evaluates

Evaluates dense retrieval models on instruction-following and standard out-of-domain tasks, specifically probing their ability to leverage natural language prompts for zero-shot hyperparameter tuning and robustness to query phrasing.

Datasets

• FollowIR — total ?; splits: test (-1)
• InstructIR — total ?; splits: test (-1)
• MS MARCO — total ?; splits: dev (-1)
• BEIR — total ?; splits: test (-1); repo https://github.com/beir-cellar/beir

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the DCG of the top-10 retrieved documents relative to ground truth relevance, normalized by the ideal DCG (IDCG) for the query.
• p-MRR — range: [0, 1]
  • Mean Reciprocal Rank for prompt-instructed queries. Averages the reciprocal of the rank of the first relevant document across all queries.
• MAP — range: [0, 1]
  • Mean Average Precision. Averages the precision at each relevant document's rank across all queries.

Input / output format

Input: Query string, optionally prefixed with a natural language instruction/prompt. The model computes relevance scores against a fixed document corpus.

Output: Ranked list of retrieved documents (or relevance scores) ordered by predicted relevance to the query.

Scoring recipe

def compute_ndcg_at_10(gold_ids, pred_ids, k=10):
    gold_set = set(gold_ids)
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(pred_ids[:k]) if doc in gold_set)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold_set), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• BEIR lacks uniform dev/train splits across its 13 subsets; prompt selection is only feasible for 7 datasets, requiring careful distinction between 'Selected Prompt' (dev-tuned) and 'Best Prompt' (oracle-tuned) reporting.
• Prompting standard retrievers (BM25, RepLLaMA) degrades performance, so prompt-based evaluation is only valid for instruction-trained models like Promptriever.

Evidence (verbatim from paper)

We sample 10 queries from each of the validation (or train if there is no validation set) to use as the prompt tuning set. We also create 10 generic prompts that could work across retrieval datasets. However, not all of the BEIR datasets have train/dev data to sample validation examples from for selecting a prompt. We thus show results in two ways (Table[4]): (1) when there is a dev set: we select the best dev prompt as the test prompt (Selected Prompt column) and leave the score blank for datasets without a dev/train set; and (2) taking the best prompt of the ten (Best Prompt column). We see in Table[4] that, for Promptriever, using the best prompt brings significant gains to BEIR average performance (+1.4 nDCG@10; gains versus no prompt for 12/13 datasets and tied on the last).

Citation

@misc{weller2024promptriever,
  title={Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
  author={Weller et al. (2024)},
  year={2024},
  note={arXiv:2409.11136}
}

• arXiv: 2409.11136