LLM Search Agent Eval

llm-search-agent-eval

Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design — Huang et al. (2025) (arXiv:2511.20048, 2025)

What this evaluates

Evaluates the end-to-end latency and answer accuracy of LLM-based search agents operating in a ReAct workflow with external Wikipedia API calls. It probes the system's ability to balance speculative action execution with verification to reduce inference time while maintaining multi-hop reasoning quality.

Datasets

• HotPotQA — total ?; splits: test (-1)
• 2WikiMultihopQA — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Exact-match accuracy: 1 if the model's final answer exactly matches the ground truth answer, 0 otherwise. Reported as a percentage across the benchmark subset.
• Latency Speedup — range: other
  • Ratio of baseline end-to-end latency to the proposed method's end-to-end latency, computed per request or averaged across request rates.
• Action Buffer Hit Rate — range: percent
  • Proportion of actions generated by the main agent path in the Verified Speculation Phase that match speculative actions already stored in the buffer. Explicitly excludes steps handled by the Aggressive Speculation Phase.

Input / output format

Input: A natural language question or query. The agent processes it via a ReAct loop, receiving tool outputs (Wikipedia search results) as intermediate inputs.

Output: A final textual answer to the question, produced after a sequence of reasoning steps and tool calls. The agent may also output intermediate speculative actions during evaluation.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Benchmarks use randomly sampled subsets rather than full test sets, limiting direct comparison with prior work.
• Latency measurements are hardware-specific (RTX 3090, A100, A800) and depend on tensor parallelism settings, making cross-platform speedup claims context-dependent.
• Action Buffer Hit Rate explicitly excludes the aggressive speculation phase, so it does not capture the full latency reduction benefit.

Evidence (verbatim from paper)

Across all evaluated models and benchmarks, SPAgent maintains accuracy on par with the baseline, and in some cases even improves it, as shown in Table[II]. Notably, the Qwen2.5-32B model exhibit accuracy gains exceeding 5% on TriviaQA, indicating that the reduced reasoning in the Aggressive Speculation Phase can occasionally enhance decision quality.

Citation

@misc{huang2025spagent,
  title={Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2511.20048}
}

• arXiv: 2511.20048