Browsecomp Plus Eval

browsecomp-plus-eval

BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent — Chen et al. (2025) (arXiv:2508.06600, 2025)

What this evaluates

Evaluates the end-to-end effectiveness of deep-research agents in retrieving evidence and answering complex queries, as well as the standalone effectiveness of various retrievers. It probes the interplay between retrieval quality, reasoning capability, and search efficiency in agentic workflows.

Datasets

• BrowseComp-Plus — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Computed via an LLM-as-judge (gpt-4.1) that compares the model's final answer against the ground truth using a fixed evaluation prompt.
• Recall — range: [0, 1]
  • Measures the fraction of human-verified evidence documents successfully retrieved by the agent across its entire interaction.
• Search Calls — range: other
  • The average number of search API invocations made per query.
• Calibration Error — range: [0, 1]
  • Measures how closely a model's predicted confidence percentage matches its actual accuracy, typically computed as Expected Calibration Error (ECE).
• Recall@k — range: [0, 1]
  • Standard TREC-style retrieval metric measuring the fraction of relevant documents retrieved in the top-k results.
• nDCG@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k, measuring ranked retrieval quality with position-based discounting.

Input / output format

Input: A natural language query and a retriever tool interface that returns the top-k (k=5) search results, each truncated to the first 512 tokens.

Output: A final answer string accompanied by a confidence estimate (percentage), and optionally cited source documents.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    # Uses gpt-4.1 as judge with a specific prompt
    return llm_as_judge_score(predictions, gold_answers)

def compute_recall(retrieved_docs, gold_docs):
    return len(set(retrieved_docs) & set(gold_docs)) / len(gold_docs)

def compute_calibration_error(confidences, accuracies):
    bins = 10
    ece = 0
    for i in range(bins):
        mask = (confidences >= i/10) & (confidences < (i+1)/10)
        if mask.sum() > 0:
            ece += mask.sum() * abs(confidences[mask].mean() - accuracies[mask].mean())
    return ece / len(confidences)

Common pitfalls

• Retrievers truncate documents to 512 tokens, which may exclude ground-truth answers for some queries (though 86.5% retain it).
• Accuracy relies on an LLM-as-judge (gpt-4.1) rather than exact match, introducing potential judge bias or inconsistency.
• Calibration error cannot be computed for Search-R1 because its fine-tuned output format lacks a confidence score.

Evidence (verbatim from paper)

We report end-to-end effectiveness of the deep research systems with four metrics: Accuracy, Recall, and Search Calls. Accuracy follows BrowseComp: an LLM-as-judge (gpt-4.1) compares the model’s final answer against the ground truth using the evaluation prompt listed in Appendix[F]. Recall measures how many human-verified evidence documents the agent retrieved during its entire interaction. Search Calls is the average number of search API invocations per query.

Citation

@misc{chen2025browsecompplus,
  title={BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.06600}
}

• arXiv: 2508.06600