Long Context Retrieval Eval

long-context-retrieval-eval

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context — Georgiev et al. (2024) (arXiv:2403.05530, 2024)

What this evaluates

Probes a model's ability to accurately retrieve hidden, specific information (needles) embedded within extremely long multimodal sequences (text, video, audio) and assesses its predictive stability over millions of tokens.

Datasets

• Paul Graham Essays (Synthetic) — total ?; splits: test (-1)
• AlphaGo Documentary — total ?; splits: test (-1)
• VoxPopuli — total ?; splits: test (-1)

Metrics

• recall (primary) — range: [0, 1]
  • Percentage of queries where the model's output correctly contains the exact hidden needle (number, word, or phrase) inserted at a specific depth in the context.
• cumulative_nll — range: [0, ∞)
  • Cumulative negative log-likelihood of tokens at each position in the sequence, measuring next-token prediction accuracy over long contexts. Lower values indicate better prediction.

Input / output format

Input: A long multimodal context (text, video frames sampled at 1fps, or audio) of variable length up to 10M tokens, containing a hidden 'needle' (e.g., a specific number, word, or phrase) inserted at a known depth, followed by a natural language query asking to retrieve the needle.

Output: A text response containing the exact hidden needle (e.g., the magic number or secret word).

Scoring recipe

def score_haystack(predictions, gold_needles):
    correct = 0
    for pred, gold in zip(predictions, gold_needles):
        if gold in pred:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Confusing relative depth (percentage) with absolute token position when analyzing retrieval performance.
• Assuming API context limits reflect model capabilities, as many baselines are tested only up to their API's max context window.
• Multimodal haystack tasks require precise frame/audio alignment; sampling rates (e.g., 1fps) drastically affect token count and retrieval difficulty.

Evidence (verbatim from paper)

We insert a needle at linearly spaced intervals from the beginning to the end of the context, where the needle is i.e., "The special magic {city} number is: {number}" where the city and number are varied for each query, and query the model to return the magic number for a specific city. We report whether the magic number recall was correct at various context lengths (x axis - the haystack) as a function of its position in the input sequence expressed in terms of depth percentage (y axis), e.g., depth at 100% would indicate a needle inserted at the very end of the input whereas 0% at the very beginning.

Citation

@misc{georgiev2024gemini15,
  title={Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context},
  author={Georgiev et al. (2024)},
  year={2024},
  note={arXiv:2403.05530}
}

• arXiv: 2403.05530