Followir Eval

followir-eval

FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions — Weller et al. (2024) (arXiv:2403.15246, 2024)

What this evaluates

Evaluates whether information retrieval models can follow complex, long-form instructions derived from TREC narratives to determine document relevance. It probes the model's ability to interpret conditional, negated, and composite relevance criteria rather than relying solely on keyword matching.

Datasets

• Robust04 — total ?; splits: test (-1)
• News21 — total ?; splits: test (-1)
• Core17 — total ?; splits: test (-1)

Metrics

• p-MRR (primary) — range: [0, 1]
  • Mean Average Precision: the mean of the average precision scores across all queries. Average precision is the precision averaged at each rank where a relevant document is retrieved.
• nDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5: measures ranking quality by summing graded relevance values scaled by the logarithm of the position rank, normalized by the ideal DCG.

Input / output format

Input: A search query paired with a long-form instruction (TREC narrative) specifying relevance criteria, and a candidate document from the collection to be scored.

Output: A relevance score or binary judgment for the candidate document, used to rank the collection according to the instruction.

Scoring recipe

For each query-instruction pair:
1. Retrieve top-k documents from the collection.
2. Re-score or re-rank documents based on whether they satisfy the instruction's relevance criteria.
3. Compute p-MRR or nDCG@5 by comparing the instruction-aware ranking against the ground truth relevance labels.
4. Report delta ($\Delta$) as the difference between instruction-following score and original baseline score.

Common pitfalls

• Evaluation instances differ per model because only the documents actually retrieved by a specific model are re-evaluated against the instruction.
• Models often fail to interpret nuanced relevance criteria and rely on keyword matching, which can cause performance degradation when instructions are added.
• Delta scores can be positive or negative, meaning instruction-following does not always improve retrieval performance compared to the baseline.

Evidence (verbatim from paper)

Table 5: FollowIRscores on the full retrieval collection (thus rerankers are not included). As the base score is different, there are different numbers of relevant documents they are being evaluated on for p-MRR. Thus, we only report the original (no-instruction) score and the delta when using the TREC instructions.

Citation

@misc{weller2024followir,
  title={FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions},
  author={Weller et al. (2024)},
  year={2024},
  note={arXiv:2403.15246}
}

• arXiv: 2403.15246