Loco1 Eval

loco1-eval

Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT — Jon Saad-Falcon et al. (arXiv:2402.07440, 2024)

What this evaluates

Evaluates long-context retrieval capabilities on real-world documents where relevant information spans entire texts, such as legal contracts and medical notes. It specifically probes a model's ability to locate and rank relevant passages without relying on truncation or chunking strategies that often bias standard retrievers.

Datasets

• LoCoV1 — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Measures ranking quality by comparing the discounted cumulative gain of the predicted ranking to the ideal ranking, normalized to [0,1]. Accounts for both position and relevance quality of retrieved items.

Input / output format

Input: A query and a long document (or concatenated passages) containing the relevant information.

Output: A ranked list of retrieved documents/passages.

Scoring recipe

def compute_ndcg_at_10(retrieved, relevant):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(retrieved[:10]))
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(sorted(relevant, reverse=True)[:10]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Assuming truncation or chunking with embedding averaging is a fair baseline for long-context retrieval; the paper shows these often fail or worsen performance.
• Ignoring position bias in long documents; models with shorter max sequence lengths drop significantly when relevant info is at the end.
• Evaluating efficiency on average chunk time instead of full-document embedding throughput.

Evidence (verbatim from paper)

We use nDCG@10 as the quality metric for LoCoV1. nDCG@10 measures the ranking quality of information retrieval systems, accounting for both the position and quality of the items in the retrieved sequence.

Citation

@misc{saadfalcon2024loco,
  title={Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT},
  author={Jon Saad-Falcon et al.},
  year={2024},
  note={arXiv:2402.07440}
}

• arXiv: 2402.07440