Medbrowsecomp Eval

medbrowsecomp-eval

MedBrowseComp: Benchmarking Medical Deep Research and Computer Use — Chen et al. (2025) (arXiv:2505.14963, 2025)

What this evaluates

Evaluates AI agents' ability to perform multi-hop, evidence-grounded medical information retrieval from live, heterogeneous web sources. It probes long-horizon web navigation, tool allocation, source verification, and the capacity to reconcile conflicting or dense biomedical data.

Datasets

• MedBrowseComp-50 — total 50; splits: test (50); repo https://github.com/shan23chen/MedBrowseComp
• MedBrowseComp-605 — total 605; splits: test (605); repo https://github.com/shan23chen/MedBrowseComp

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly answered questions out of the total. A half-point is awarded for specific 2-hop questions where the model retrieves a sub-entity's company name instead of the final answer.

Input / output format

Input: A multi-hop medical information-seeking question requiring navigation of live web sources, with access to agentic tools (e.g., web search, browser, RAG/memory).

Output: A direct answer to the question, often requiring specific formatting (e.g., MM-DD-YYYY, YYYY, stock ticker) and verifiable source citations.

Scoring recipe

def compute_accuracy(predictions, gold, total_questions):
    score = 0
    for pred, gold_ans in zip(predictions, gold):
        if exact_match(pred, gold_ans):
            score += 1.0
        elif is_partial_2hop_match(pred, gold_ans):
            score += 0.5
    return score / total_questions

Common pitfalls

• Agents exhaust tool-call quotas on preliminary tasks, leaving final critical queries unanswered.
• Models cite secondary press releases instead of authoritative sources (e.g., FDA Orange Book), leading to incorrect dates or claims.
• Agents fail to robustly parse dense tables in multi-page PDFs, extracting only partial milestones.
• Greedy attribute extraction causes models to copy the first matching date/field encountered rather than the correct structured metadata.

Evidence (verbatim from paper)

Figure [3] summarizes accuracy on MedBrowseComp-50. Across all systems, performance decays monotonically with hop count, corroborating prior evidence that long-horizon web navigation remains an open challenge for frontier LLM agents. Nevertheless, deep research variants–agents that allow iterative browsing steps rather than a single query—had improved performance. For example, O3 deepresearch answers 25.5/50 questions correctly, a 34% relative gain over O3 search (19/50); Gemini-2.5-pro deepsearch shows 75% improvement over its single-shot analogue (24.5 vs. 14). These gains are most pronounced on the hardest 4- and 5-hop splits, where deep research agents more than double the baseline accuracy. ... Half point given to a specific 2-hop question where the model retrieved the sub-entities’ company name instead.

Citation

@misc{chen2025medbrowsecomp,
  title={MedBrowseComp: Benchmarking Medical Deep Research and Computer Use},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.14963}
}

• arXiv: 2505.14963