Long Context Reasoning Eval

long-context-reasoning-eval

LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts — Siyuan Wang et al. (2025) (arXiv:2510.19363, 2025)

What this evaluates

Evaluates a model's ability to perform multi-hop question answering and information retrieval over extremely long contexts (up to 128K tokens), while also measuring preservation of short-context reasoning and instruction-following capabilities.

Datasets

• LongBench v1 — total ?; splits: test (-1)
• LongBench v2 — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• MATH-500 — total ?; splits: test (-1)
• IFEval — total ?; splits: test (-1)
• Needle in a Haystack — total ?; splits: test (-1)
• RULER — total ?; splits: test (-1)

Metrics

• pass@1 accuracy (primary) — range: percent
  • For each problem, the model generates 8 candidate responses. The metric is the fraction of problems where at least one candidate contains the correct answer, reported as a percentage.

Input / output format

Input: A multi-hop question paired with a long-context document or set of documents (up to 128K tokens).

Output: A reasoning trace followed by a final answer. Inference samples 8 responses per problem with temperature 0.6 and max 10K output tokens.

Scoring recipe

correct_count = 0
for problem in dataset:
    responses = model.generate(problem.context, problem.question, n=8, temperature=0.6)
    if any(extract_answer(r) == problem.gold_answer for r in responses):
        correct_count += 1
return (correct_count / len(dataset)) * 100

Common pitfalls

• The paper uses a custom 'two-way substring exact match' verifier for RL training rewards, but evaluation uses standard pass@1 accuracy; do not confuse the two.
• Baselines may use single-pass inference, while LoongRL explicitly samples 8 solutions per problem to compute pass@1; ensure fair comparison by matching sampling strategies.
• Context length generalization is tested up to 128K tokens despite training only on 16K inputs; results degrade sharply for baselines at longer lengths.

Evidence (verbatim from paper)

For inference, reasoning models and our models use temperature 0.6, with up to 128K input tokens and 10K output tokens. We sample eight solutions per problem and report average pass@1 accuracy.

Citation

@misc{wang2025loongrl,
  title={LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts},
  author={Siyuan Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.19363}
}

• arXiv: 2510.19363