Infoseek Eval

infoseek-eval

Open Data Synthesis For Deep Research — Xia et al. (arXiv:2509.00375, 2025)

What this evaluates

Evaluates LLMs on complex, multi-step reasoning and agentic search tasks, including single-hop and multi-hop question answering as well as deep research benchmarks requiring web search and synthesis.

Datasets

• NQ — total ?; splits: test (-1)
• TQA — total ?; splits: test (-1)
• PopQA — total ?; splits: test (-1)
• HQA — total ?; splits: test (-1)
• 2Wiki — total ?; splits: test (-1)
• MSQ — total ?; splits: test (-1)
• Bamb — total ?; splits: test (-1)
• BrowseComp-Plus — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of instances where the model's final extracted answer matches the ground truth.
• Search Calls — range: other
  • Average number of search engine invocations per task instance.

Input / output format

Input: Question or research task prompt requiring multi-step reasoning and web search.

Output: Model generates responses enclosed in specific tags: for reasoning, for queries, for retrieved summaries, and for the final response.

Scoring recipe

def compute_metrics(predictions, gold):
    correct = 0
    search_calls = 0
    for pred, gold_ans in zip(predictions, gold):
        extracted = extract_tag(pred, 'answer')
        if exact_match(extracted, gold_ans):
            correct += 1
        search_calls += count_tag(pred, 'search')
    return {'Accuracy': correct / len(predictions) * 100, 'Search Calls': search_calls / len(predictions)}

Common pitfalls

• Models must strictly follow the XML-like tag structure (, , ); failure to do so breaks answer extraction.
• High Accuracy does not imply efficient search usage; Search Calls are reported separately and can vary significantly across models.
• Benchmarks like BrowseComp-Plus involve noisy web results, where models often lose focus or hallucinate without proper intermediate summarization.

Evidence (verbatim from paper)

Table 4: Model performance on the BrowseComp-Plus benchmark for complex reasoning tasks.

R = 1 if format and extracted answer are both correct, 0 otherwise

Citation

@misc{xia2025opendata,
  title={Open Data Synthesis For Deep Research},
  author={Xia et al.},
  year={2025},
  note={arXiv:2509.00375}
}

• arXiv: 2509.00375