Ssp Eval

ssp-eval

Search Self-play: Pushing the Frontier of Agent Capability without Supervision — Lu et al. (2025) (arXiv:2510.18821, 2025)

What this evaluates

Evaluates the capability of deep search agents to answer complex factual and multi-hop questions using retrieval-augmented generation and multi-turn reasoning. It probes the agent's ability to dynamically adjust search strategies, verify information via RAG, and synthesize accurate answers under constrained tool-use budgets.

Datasets

• NQ — total 500; splits: test (500)
• TriviaQA — total 500; splits: test (500)
• PopQA — total 500; splits: test (500)
• HotpotQA — total 500; splits: test (500)
• 2WikiMultiHopQA — total 500; splits: test (500)
• Musique — total 500; splits: test (500)
• Bamboogle — total 125; splits: test (125)

Metrics

• pass@1 accuracy (primary) — range: [0, 1]
  • LLM-as-a-judge evaluation where Qwen2.5-32B-Instruct assesses whether the agent's final answer is correct relative to the ground truth. pass@1 denotes the fraction of questions answered correctly on the first attempt.

Input / output format

Input: Natural language question/query.

Output: Final answer string generated by the agent after up to 10 rounds of tool-integrated search.

Scoring recipe

def score(predictions, gold_answers):
    judge = load_llm('Qwen2.5-32B-Instruct')
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        prompt = f'Question: {pred["question"]}\nPredicted Answer: {pred["answer"]}\nGround Truth: {gold}\nIs the predicted answer correct? Answer YES or NO.'
        response = judge.generate(prompt)
        if 'YES' in response.upper():
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Evaluators must sample exactly 500 QA pairs per benchmark (except Bamboogle which uses all 125) to match the paper's protocol.
• The LLM-as-a-judge metric relies on Qwen2.5-32B-Instruct; using a different judge model or prompt may yield different pass@1 scores.
• Search tool calls are strictly limited to 10 rounds per trajectory; exceeding this limit violates the evaluation constraints.

Evidence (verbatim from paper)

Following recent work (Gao et al., 2025a), we adopt LLM-as-a-judge as standard metric for evaluation. Qwen2.5-32B-Instruct (Yang et al., 2024a) is deployed as the judge model. All results are reported in terms of pass@1 accuracy. Following the practice in prior works, we randomly sample 500 question-answer (QA) pairs on each benchmark to reduce the evaluation overhead while maintaining statistical reliability. For Bamboogle (Press et al., 2022), all 125 test samples are used for evaluation.

Citation

@misc{lu2025searchselfplay,
  title={Search Self-play: Pushing the Frontier of Agent Capability without Supervision},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2510.18821}
}

• arXiv: 2510.18821