Helmet Long Context Eval

helmet-long-context-eval

Self-Taught Agentic Long Context Understanding — Zhuang et al. (2025) (arXiv:2502.15920, 2025)

What this evaluates

Evaluates a model's ability to retain, process, and reason over extended contexts (8K to 128K tokens) across retrieval-augmented generation (RAG) and long-range question answering (LongQA) tasks. It probes robustness to noise, multi-hop reasoning, and memorization in long-context settings.

Datasets

• HELMET — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of questions where the model's generated answer is judged correct by GPT-4o. Calculated as the number of correct predictions divided by the total number of questions.

Input / output format

Input: Question paired with relevant contexts, with irrelevant context appended to pad input length to 8K, 16K, 32K, 64K, or 128K tokens. For LongQA tasks, the input is truncated to the first N tokens, potentially excluding the relevant context.

Output: Free-form text answer generated by the model.

Scoring recipe

correct = 0
for q, gold, pred in dataset:
    prompt = f'Question: {q}\nGold: {gold}\nModel: {pred}\nIs the model answer correct?'
    if gpt4o_judge(prompt).lower().startswith('yes'):
        correct += 1
return correct / len(dataset)

Common pitfalls

• GPT-4o judge prompt template is located in Appendix F and must be used verbatim to ensure consistent scoring.
• LongQA tasks may show artificial performance gains at longer context lengths simply because the necessary information becomes available in the untruncated input, unlike RAG tasks where noise increases.
• Short-context benchmark scores (ARC, GSM8K, MMLU, etc.) are only reported as an aggregate average in the main table; individual task scores are in Appendix B.

Evidence (verbatim from paper)

In this section, we assess our method AgenticLU using a suite of evaluation tasks drawn from the HELMET long-context benchmark. Our experiments focus on testing models’ ability to retain, process, and reason over extended contexts ranging from 8K to 128K tokens. We use GPT-4o as the judge for answer correctness, with the prompt template shown in[appendix F]. We report accuracies for all datasets.

Citation

@misc{zhuang2025agenticlu,
  title={Self-Taught Agentic Long Context Understanding},
  author={Zhuang et al. (2025)},
  year={2025},
  note={arXiv:2502.15920}
}

• arXiv: 2502.15920