# Long Context Retrieval Eval

> Probes a model's ability to accurately retrieve hidden, specific information (needles) embedded within extremely long multimodal sequences (text, video, audio) and assesses its predictive stability over millions of tokens. Use when the user wants to benchmark on Paul Graham Essays (Synthetic), AlphaGo Documentary, VoxPopuli, or asks about evaluating this task. Reports recall.

- Skill: `qhjqhj00/long-context-retrieval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/long-context-retrieval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/long-context-retrieval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/long-context-retrieval-eval

---


# long-context-retrieval-eval

> Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context — Georgiev et al. (2024) (arXiv:2403.05530, 2024)

## What this evaluates

Probes a model's ability to accurately retrieve hidden, specific information (needles) embedded within extremely long multimodal sequences (text, video, audio) and assesses its predictive stability over millions of tokens.

## Datasets

- **Paul Graham Essays (Synthetic)** — total ?; splits: test (-1)
- **AlphaGo Documentary** — total ?; splits: test (-1)
- **VoxPopuli** — total ?; splits: test (-1)

## Metrics

- `recall` **(primary)** — range: [0, 1]
  - Percentage of queries where the model's output correctly contains the exact hidden needle (number, word, or phrase) inserted at a specific depth in the context.
- `cumulative_nll` — range: [0, ∞)
  - Cumulative negative log-likelihood of tokens at each position in the sequence, measuring next-token prediction accuracy over long contexts. Lower values indicate better prediction.

## Input / output format

**Input**: A long multimodal context (text, video frames sampled at 1fps, or audio) of variable length up to 10M tokens, containing a hidden 'needle' (e.g., a specific number, word, or phrase) inserted at a known depth, followed by a natural language query asking to retrieve the needle.

**Output**: A text response containing the exact hidden needle (e.g., the magic number or secret word).

## Scoring recipe

```python
def score_haystack(predictions, gold_needles):
    correct = 0
    for pred, gold in zip(predictions, gold_needles):
        if gold in pred:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Confusing relative depth (percentage) with absolute token position when analyzing retrieval performance.
- Assuming API context limits reflect model capabilities, as many baselines are tested only up to their API's max context window.
- Multimodal haystack tasks require precise frame/audio alignment; sampling rates (e.g., 1fps) drastically affect token count and retrieval difficulty.

## Evidence (verbatim from paper)

> We insert a needle at linearly spaced intervals from the beginning to the end of the context, where the needle is i.e., "The special magic {city} number is: {number}" where the city and number are varied for each query, and query the model to return the magic number for a specific city. We report whether the magic number recall was correct at various context lengths (x axis - the haystack) as a function of its position in the input sequence expressed in terms of depth percentage (y axis), e.g., depth at 100% would indicate a needle inserted at the very end of the input whereas 0% at the very beginning.

## Citation

```bibtex
@misc{georgiev2024gemini15,
  title={Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context},
  author={Georgiev et al. (2024)},
  year={2024},
  note={arXiv:2403.05530}
}
```

- arXiv: 2403.05530