Document Haystack Eval

document-haystack-eval

Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark — Huybrechts et al. (2025) (arXiv:2507.15882, 2025)

What this evaluates

This benchmark evaluates Vision Language Models' ability to retrieve specific textual or multimodal "needles" embedded within long documents ranging from 5 to 200 pages. It probes visual-text alignment, long-context retrieval capabilities, and performance degradation as document length and token consumption increase.

Datasets

• Document Haystack — total 400; splits: test (-1); repo https://github.com/amazon-science/document-haystack

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly retrieved needles out of the total evaluated. Calculated as (number of correct retrievals / total needles) * 100.

Input / output format

Input: Long documents (5–200 pages) provided either as images or parsed text, accompanied by a natural language query asking to retrieve a specific embedded needle (e.g., “What is the secret fruit in the document?”).

Output: Extracted text or description corresponding to the queried needle. Exact formatting is not strictly enforced, but retrieval must match the ground truth needle.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• API token limits cause models like GPT-4o-mini to fail or truncate on documents exceeding 50 pages, artificially lowering reported accuracy.
• Preprocessing method (converting PDFs to images vs. extracting raw text) drastically changes performance; comparing models that use different preprocessing pipelines directly is misleading.
• Token compression strategies vary widely across models (e.g., Nova Lite uses ~6x more tokens per image than Gemini), which impacts context window utilization and retrieval accuracy independently of model capability.

Evidence (verbatim from paper)

The table shows the retrieval accuracy across different document lengths of three API providers: Nova Lite, Gemini Flash-2.0, and GPT-4o-mini.

Citation

@misc{huybrechts2025documenthaystack,
  title={Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark},
  author={Huybrechts et al. (2025)},
  year={2025},
  note={arXiv:2507.15882}
}

• arXiv: 2507.15882