Ifir Eval

ifir-eval

IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval — Song et al. (2025) (arXiv:2503.04644, 2025)

What this evaluates

This benchmark evaluates an information retrieval system's ability to follow complex, domain-specific instructions when retrieving relevant passages. It probes whether models can interpret nuanced constraints (e.g., patient demographics, legal case details, financial goals) rather than just matching keyword semantics.

Datasets

• IfIR — total 2426; splits: finance (1718), scientific (152), literature (86), law (253), healthcare (215); repo https://github.com/SighingSnow/IFIR

Metrics

• nDCG@20 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20. Standard IR metric measuring the quality of the ranked list of retrieved passages against binary relevance labels.
• InstFol@20 (primary) — range: percent
  • An LLM-based evaluation metric that scores the top-20 retrieved passages on their ability to satisfy the complex, domain-specific instructions attached to each query. Scores are aggregated and reported as a percentage.

Input / output format

Input: A query augmented with a domain-specific instruction (varying in complexity from simple to highly detailed) and a corpus of candidate passages.

Output: A ranked list of retrieved passages (top-20).

Scoring recipe

def compute_ndcg_at_20(predictions, gold):
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate(predictions[:20]))
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(sorted(gold, reverse=True)[:20]))
    return dcg / idcg if idcg > 0 else 0.0

def compute_instfol_at_20(predictions, instruction):
    scores = []
    for passage in predictions[:20]:
        score = llm_judge.evaluate(passage, instruction) # Returns 1-5
        scores.append(score)
    return (sum(scores) / (20 * 5)) * 100

Common pitfalls

• Assuming LLM-based retrievers will automatically outperform lexical models like BM25 on InstFol; the paper explicitly notes BM25 often wins due to better glossary alignment with domain-specific instructions.
• Ignoring the three-tier instruction complexity levels (simple, demographic/contextual, goal-specific), which significantly impact model performance and should be analyzed separately.
• Treating InstFol as a standard relevance metric; it specifically measures instruction-following compliance, not just topical relevance.

Evidence (verbatim from paper)

Table 3: Performance of retrievers on IfIR measured by nDCG@20 and InstFol@20 (%). ... Specifically, we first use LLM (i.e., GPT-4o) to assess the relevance of each original relevant passage to the instruction. The LLM is tasked with generating justification explanations alongside its relevance assessments... Human annotators then review the relevance of each passage and the justifications provided by the LLM.

Citation

@misc{song2025ifir,
  title={IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2503.04644}
}

• arXiv: 2503.04644