Longembed Eval

longembed-eval

LongEmbed: Extending Embedding Models for Long Context Retrieval — Zhu et al. (2024) (arXiv:2404.12096, 2024)

What this evaluates

Evaluates embedding models' ability to retrieve relevant information from long contexts (up to 32k tokens) and compares the effectiveness of various context window extension strategies. It also probes the extrapolation capabilities of Absolute Positional Encoding (APE) versus Rotary Positional Encoding (RoPE) in retrieval tasks.

Datasets

• LongEmbed — total ?; splits: Passkey (-1), Needle (-1), NarrativeQA (-1), QMSum (-1), SummScreenFD (-1), 2WikiMultihopQA (-1); repo https://github.com/dwzhu-pku/LongEmbed

Metrics

• accuracy (%) (primary) — range: percent
  • Percentage of correctly retrieved or answered instances per subset, averaged across all six subsets (Passkey, Needle, NarrativeQA, QMSum, SummScreenFD, 2WikiMultihopQA).

Input / output format

Input: A query paired with a long context document (ranging from 512 to 32,768 tokens) containing the target information or answer.

Output: Dense embedding vectors for the query and the context document, used for cosine similarity-based retrieval.

Scoring recipe

def compute_longembed_score(predictions, golds):
    subset_scores = []
    for subset in ['Passkey', 'Needle', 'NarrativeQA', 'QMSum', 'SummScreenFD', '2WikiMultihopQA']:
        correct = sum(1 for pred, gold in zip(predictions[subset], golds[subset]) if pred == gold)
        subset_scores.append(correct / len(golds[subset]) * 100)
    return sum(subset_scores) / len(subset_scores)

Common pitfalls

• Models trained on data overlapping with the LongEmbed test set (e.g., M2*) must be excluded to avoid data contamination.
• Plug-and-play extension methods (e.g., PI, GP, PCW) should not be confused with full model fine-tuning; only position embeddings are typically modified or frozen during extension.
• Comparing APE and RoPE models requires controlling for identical training data and procedures, otherwise performance gaps may stem from pretraining differences rather than positional encoding.

Evidence (verbatim from paper)

Table 2 demonstrates the performance of existing embedding models on our LongEmbed benchmark. ... Table 3: Results (%) of context window extension methods on E5-RoPEBase and E5-Mistral. For datasets, P, N, NQA, QMS, SFD, WQA is short for Passkey, Needle, NarrativeQA, QMSum, SummScreenFD, 2WikiMultihopQA.

Citation

@misc{zhu2024longembed,
  title={LongEmbed: Extending Embedding Models for Long Context Retrieval},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2404.12096}
}

• arXiv: 2404.12096