Deepwidesearch Eval

deepwidesearch-eval

DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking — Lan et al. (2025) (arXiv:2510.20168, 2025)

What this evaluates

Evaluates agentic systems' ability to perform wide-scale information collection and deep multi-hop reasoning simultaneously to fill structured result tables. It probes combinatorial search complexity, tool orchestration, reflection, and context management in real-world information-seeking tasks.

Datasets

• DeepWideBenchmark — total 220; splits: test (-1)

Metrics

• Success Rate (primary) — range: percent
  • The percentage of queries where the agent's final output exactly matches the gold-standard table structure and content. Evaluated as Avg@4 (average over 4 attempts) and Pass@4 (best-of-4 success).
• Row F1 Score — range: percent
  • Harmonic mean of precision and recall computed over table rows, measuring how well the agent groups related entities together.
• Item F1 Score — range: percent
  • Harmonic mean of precision and recall computed over individual table items/cells, measuring fine-grained entity extraction accuracy.
• Column F1 — range: percent
  • Harmonic mean of precision and recall computed over table columns, measuring the agent's ability to consistently populate specific attribute categories across all rows.
• Core Entity Accuracy (CE Acc.) — range: percent
  • Exact match rate of the most critical entities required by the query, regardless of table formatting or peripheral details.

Input / output format

Input: Natural language questions/tasks requiring multi-hop reasoning and broad information collection. Agents receive the task and have access to Google Search API and Webpage Visit tools. HTML content is pre-summarized by the agent's backbone LLM before processing.

Output: Markdown-formatted tables containing the requested information, structured with rows, items, and columns corresponding to the query requirements.

Scoring recipe

def score(predictions, gold):
    # Success Rate: exact match over 4 attempts
    success = 1.0 if predictions == gold else 0.0
    # Parse markdown tables into structured lists
    pred_rows = parse_table(predictions, axis='row')
    gold_rows = parse_table(gold, axis='row')
    row_f1 = f1_score(pred_rows, gold_rows)
    item_f1 = f1_score(parse_table(predictions, axis='item'), parse_table(gold, axis='item'))
    col_f1 = f1_score(parse_table(predictions, axis='col'), parse_table(gold, axis='col'))
    # Core Entity Accuracy: exact set match
    ce_acc = 1.0 if set(predictions.core_entities) == set(gold.core_entities) else 0.0
    return {'success_rate': success, 'row_f1': row_f1, 'item_f1': item_f1, 'col_f1': col_f1, 'ce_acc': ce_acc}

Common pitfalls

• Agents frequently fail due to brittle output formatting, producing invalid markdown tables instead of correctly structured results.
• Overreliance on internal knowledge leads to poor retrieval coverage, while insufficient reasoning before tool calls restricts query precision.
• Context overflow and inference errors during multi-step tool orchestration cause incomplete task solving, especially for models like Gemini 2.5 Pro.

Evidence (verbatim from paper)

The complete results are presented in Table [2]. It can be found that most baselines demonstrate near-zero success rates, with only WebSailor (Gemini 2.5 Pro) and WebSailor (Claude Sonnet 4) exceeding 1-2% in Success Rate (Avg@4), confirming the inherent complexity of simultaneously handling deep reasoning and wide-scale information collection.

Citation

@misc{lan2025deepwidesearch,
  title={DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking},
  author={Lan et al. (2025)},
  year={2025},
  note={arXiv:2510.20168}
}

• arXiv: 2510.20168