qhjqhj00 deep-research-accuracy-eval
OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis — Li et al. (2026) (arXiv:2603.20278, 2026)
What this evaluates
Evaluates an agent's ability to perform long-horizon, multi-step web research to answer complex factual questions. It probes the model's capacity for iterative search, evidence aggregation, and adaptive reasoning under both reproducible offline constraints and live web environments.
Datasets
- BrowseComp-Plus — total ?; splits: test (-1)
- BrowseComp — total ?; splits: test (-1)
- GAIA — total ?; splits: test (-1)
- xbench-DeepSearch — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Calculated as the percentage of questions for which the agent's final generated answer matches the ground truth. Also reported as Pass@k, measuring the fraction of questions solved by at least one of k sampled trajectories.
Input / output format
Input: A natural language research question. For BrowseComp-Plus, the agent accesses a fixed 15M-document offline corpus via a FAISS index. For other benchmarks, the agent uses the Serper API for live web search.
Output: A final textual answer to the research question.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if is_match(p, g))
return (correct / len(gold)) * 100
def compute_pass_at_k(predictions_per_q, k):
solved = sum(1 for preds in predictions_per_q if any(is_match(p, gold) for p in preds[:k]))
return (solved / len(predictions_per_q)) * 100
Common pitfalls
- Confusing the search environment: BrowseComp-Plus uses a fixed offline corpus with a FAISS index, while BrowseComp, GAIA, and xbench-DeepSearch rely on live Serper API calls.
- Assuming longer trajectories always yield better answers: Performance plateaus after ~100 turns, and failed trajectories often use more tools inefficiently rather than correctly.
- Overlooking Pass@k vs Pass@1: Many questions are solvable but only along specific reasoning paths, so evaluating only the first trajectory underestimates capability.
Evidence (verbatim from paper)
Our OpenResearcher-30B-A3B achieves 54.8% accuracy on this benchmark, substantially outperforming strong proprietary baselines including GPT-4.1 (36.4%), Claude-4-Opus (36.8%), and DeepSeek-R1 (16.4%).
Citation
@misc{li2026openresearcher,
title={OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis},
author={Li et al. (2026)},
year={2026},
note={arXiv:2603.20278}
}
- arXiv: 2603.20278