Instructir Eval

instructir-eval

INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models — Hanseok Oh et al. (2024) (arXiv:2402.14334, 2024)

What this evaluates

Evaluates whether information retrieval models can accurately follow instance-specific, user-aligned instructions rather than generic task descriptions. It probes the robustness of retrievers to instruction variations and their ability to adapt to real-world search scenarios with diverse user contexts.

Datasets

• InstructIR — total 9906; splits: test (9906); repo https://github.com/kaistAI/InstructIR

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Standard IR metric measuring ranking quality based on graded relevance of retrieved documents, normalized by the ideal DCG.
• Robustness@10 — range: [0, 1]
  • Measures instruction adherence under instruction variation. Calculated by evaluating retrieval performance across varied phrasings of the same user intent to assess consistency and resistance to lexical bias.

Input / output format

Input: A user-aligned instruction (query) and a candidate document corpus.

Output: A ranked list of top-10 documents from the corpus.

Scoring recipe

def compute_metrics(predictions, gold, k=10):
    # predictions: list of doc IDs ranked by model
    # gold: list of relevant doc IDs
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate([1 if d in gold else 0 for d in predictions[:k]]))
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(sorted([1 if d in gold else 0 for d in gold], reverse=True)[:k]))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    # Robustness@10 is computed by averaging nDCG@10 across instruction variations for the same intent
    robustness = ndcg  # Simplified; actual implementation averages across variations
    return {'nDCG@10': ndcg, 'Robustness@10': robustness}

Common pitfalls

• Models fine-tuned on task-style instructions often overfit and underperform on free-form, user-aligned instructions.
• Lexical matching models (e.g., BM25) suffer significant drops in Robustness@10 due to keyword bias rather than semantic understanding.
• Evaluation is strictly zero-shot; models are not fine-tuned on the InstructIR test set.

Evidence (verbatim from paper)

The largest model, RepLLaMa with 7B parameters, achieve the highest nDCG@10 of 87.62 and Robustness@10 of 52.58, indicating a strong correlation between model size and performance metrics in non-instruction-tuned settings.

Citation

@misc{oh2024instructir,
  title={INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models},
  author={Hanseok Oh et al. (2024)},
  year={2024},
  note={arXiv:2402.14334}
}

• arXiv: 2402.14334