Crumb Eval

crumb-eval

Benchmarking Information Retrieval Models on Complex Retrieval Tasks — Killingback et al. (2025) (arXiv:2509.07253, 2025)

What this evaluates

Evaluates information retrieval models on complex, multi-aspect, and logically structured queries across eight diverse domains. It probes the model's ability to handle nuanced document alignments, set-based operations, and context-rich instructions beyond simple keyword matching.

Datasets

• CRUMB — total ?; splits: test (-1), val (-1); repo https://github.com/jfkback/crumb

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of graded relevance scores of retrieved documents up to position 10, normalized by the ideal DCG@10.
• R@100 — range: [0, 1]
  • Recall at 100. The proportion of relevant documents retrieved within the top 100 results out of the total number of relevant documents.

Input / output format

Input: Query text (optionally prefixed with an instruction like 'Instruct: \nQuery: '). Document corpus provided as either full markdown-formatted documents or contextualized chunks (≤512 BERT tokens, prepended with hierarchical header paths).

Output: Ranked list of retrieved documents or chunks, typically returned as a list of document IDs or text snippets ordered by relevance score.

Scoring recipe

def compute_metrics(retrieved_ids, relevant_ids, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, rid in enumerate(retrieved_ids[:k]) if rid in relevant_ids)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), k)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    recall = min(len(set(retrieved_ids[:100]) & set(relevant_ids)), len(relevant_ids)) / max(len(relevant_ids), 1)
    return {'nDCG@10': ndcg, 'R@100': recall}

Common pitfalls

• Using newline-based chunking instead of the provided contextualized chunking strategy, which strips hierarchical headers and degrades performance.
• Ignoring the MaxP recommendation for passage-level evaluation, which can unfairly penalize models that retrieve relevant full documents but miss specific chunks.
• Applying LLM query rewriting uniformly, as it improves weaker models but degrades performance on strong baselines.

Evidence (verbatim from paper)

The benchmark reveals that even top retrieval models achieve only average nDCG@10 of 0.346 and R@100 of 0.587, with poor performance in low-overlap scenarios and set-based logical queries.

Citation

@misc{killingback2025crumb,
  title={Benchmarking Information Retrieval Models on Complex Retrieval Tasks},
  author={Killingback et al. (2025)},
  year={2025},
  note={arXiv:2509.07253}
}

• arXiv: 2509.07253