Wide Search Eval

wide-search-eval

WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning — Xu et al. (2026) (arXiv:2602.04634, 2026)

What this evaluates

Evaluates a model's ability to perform broad information seeking by decomposing complex queries into parallel subtasks and producing structured tabular outputs. It also measures robustness on standard single-hop and multi-hop open-domain QA tasks.

Datasets

• WideSearch — total 200; splits: test (200)

Metrics

• item F1 score (primary) — range: [0, 1]
  • Measures the overlap between predicted and gold table cells/entries. Row F1 measures row-level exact match. Success Rate (SR) indicates whether the entire task output is correct. Each task is sampled 4 times; Avg@4 averages the metric across 4 runs, Max@4 takes the highest F1 across runs, and Pass@4 for SR indicates if at least one run passes.

Input / output format

Input: Natural language query (English or Chinese) for WideSearch; standard open-domain QA questions for the QA suite.

Output: Structured tabular output for WideSearch; free-form QA answers for standard benchmarks.

Scoring recipe

# Pseudo-code for WideSearch evaluation
item_f1s, row_f1s, srs = [], [], []
for _ in range(4):
    pred = model.generate(query)
    item_f1s.append(compute_item_f1(pred, gold_table))
    row_f1s.append(compute_row_f1(pred, gold_table))
    srs.append(1.0 if pred == gold_table else 0.0)
avg_f1 = mean(item_f1s)
max_f1 = max(row_f1s)
pass_sr = 1.0 if any(srs) else 0.0

Common pitfalls

• Failing to sample each task exactly 4 times and correctly applying Avg@4, Max@4, and Pass@4 aggregation rules.
• Confusing width scaling (parallel subagents per turn) with depth scaling (sequential turns) when plotting performance vs test-time compute.
• Using online search tools instead of the specified offline Wiki2018 knowledge base, which breaks the fair comparison setup.

Evidence (verbatim from paper)

We evaluate WideSeek-R1-4B on the WideSearch benchmark to show the effectiveness of our multi-agent system trained via MARL for broad information seeking. The benchmark consists of 200 tasks, with 100 English and 100 Chinese queries requiring tabular output. We report item F1 score, row F1 score, and Success Rate (SR). Each task is sampled four times, and we report Avg@4 for all metrics, Max@4 for F1 scores, and Pass@4 for SR.

Citation

@misc{xu2026wideseekr1,
  title={WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2602.04634}
}

• arXiv: 2602.04634