# LLM Search Agent Eval

> Evaluates the end-to-end latency and answer accuracy of LLM-based search agents operating in a ReAct workflow with external Wikipedia API calls. It probes the system's ability to balance speculative action execution with verification to reduce inference time while maintaining multi-hop reasoning quality. Use when the user wants to benchmark on HotPotQA, 2WikiMultihopQA, TriviaQA, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/llm-search-agent-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/llm-search-agent-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/llm-search-agent-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/llm-search-agent-eval

---


# llm-search-agent-eval

> Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design — Huang et al. (2025) (arXiv:2511.20048, 2025)

## What this evaluates

Evaluates the end-to-end latency and answer accuracy of LLM-based search agents operating in a ReAct workflow with external Wikipedia API calls. It probes the system's ability to balance speculative action execution with verification to reduce inference time while maintaining multi-hop reasoning quality.

## Datasets

- **HotPotQA** — total ?; splits: test (-1)
- **2WikiMultihopQA** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Exact-match accuracy: 1 if the model's final answer exactly matches the ground truth answer, 0 otherwise. Reported as a percentage across the benchmark subset.
- `Latency Speedup` — range: other
  - Ratio of baseline end-to-end latency to the proposed method's end-to-end latency, computed per request or averaged across request rates.
- `Action Buffer Hit Rate` — range: percent
  - Proportion of actions generated by the main agent path in the Verified Speculation Phase that match speculative actions already stored in the buffer. Explicitly excludes steps handled by the Aggressive Speculation Phase.

## Input / output format

**Input**: A natural language question or query. The agent processes it via a ReAct loop, receiving tool outputs (Wikipedia search results) as intermediate inputs.

**Output**: A final textual answer to the question, produced after a sequence of reasoning steps and tool calls. The agent may also output intermediate speculative actions during evaluation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Benchmarks use randomly sampled subsets rather than full test sets, limiting direct comparison with prior work.
- Latency measurements are hardware-specific (RTX 3090, A100, A800) and depend on tensor parallelism settings, making cross-platform speedup claims context-dependent.
- Action Buffer Hit Rate explicitly excludes the aggressive speculation phase, so it does not capture the full latency reduction benefit.

## Evidence (verbatim from paper)

> Across all evaluated models and benchmarks, SPAgent maintains accuracy on par with the baseline, and in some cases even improves it, as shown in Table[II]. Notably, the Qwen2.5-32B model exhibit accuracy gains exceeding 5% on TriviaQA, indicating that the reduced reasoning in the Aggressive Speculation Phase can occasionally enhance decision quality.

## Citation

```bibtex
@misc{huang2025spagent,
  title={Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2511.20048}
}
```

- arXiv: 2511.20048