Needlebench Eval

needlebench-eval

NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities — Li et al. (2024) (arXiv:2407.11963, 2024)

What this evaluates

Evaluates large language models' ability to retrieve specific information and perform complex multi-point reasoning within long-context documents. It probes both information-sparse retrieval and information-dense reasoning (Ancestral Trace Challenge) across 32K and 128K token contexts.

Datasets

• NeedleBench — total ?; splits: test (-1); repo https://github.com/open-compass/opencompass

Metrics

• Overall (primary) — range: percent
  • Mean accuracy across all sub-tasks (Single-Retrieval, Multi-Retrieval, Multi-Reasoning) for a given context length and language.

Input / output format

Input: Long-context documents (32K or 128K tokens) containing synthetic 'needles' (key facts) interspersed with irrelevant filler text, followed by a natural language query requiring either direct retrieval or multi-step reasoning.

Output: Free-text answer generated by the model in response to the query.

Scoring recipe

def compute_score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
# For ATC, scores are averaged across needle counts (2, 4, ..., 512) to produce a Weighted Score.

Common pitfalls

• Models often exhibit an 'under-thinking' phenomenon, prematurely terminating reasoning despite sufficient context.
• Performance varies significantly between English and Chinese, with English typically scoring higher due to pretraining data distribution.
• Scaling parameter count does not guarantee reasoning improvements; architecture and training strategies heavily influence multi-needle performance.

Evidence (verbatim from paper)

Overall denotes the mean score across all tasks. Bold denotes the best score among all models, and underline denotes the best score under the same model scale.

Citation

@misc{li2024needlebench,
  title={NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2407.11963}
}

• arXiv: 2407.11963