qhjqhj00 voicebench-qa-eval
AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks — Maben et al. (2025) (arXiv:2506.23049, 2025)
What this evaluates
Evaluates an agent's reasoning and tool-use capabilities in spoken question answering by requiring it to process audio queries, optionally perform web searches, and generate accurate responses.
Datasets
- OpenBookQA — total ?; splits: test (-1)
- AlpacaEval — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.
score (1-5)— range: other- Open-ended response quality rated on a 1 to 5 scale by GPT-4o-mini.
Input / output format
Input: Audio query (spoken question) transcribed by ASR, passed to LLM agent with tool-use prompts.
Output: Text response (or audio via TTS) containing the answer.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return (correct / len(gold)) * 100
def compute_alpaca_score(predictions):
# Scored externally by GPT-4o-mini on 1-5 scale
return average_gpt4o_scores(predictions)
Common pitfalls
- ASR transcription errors can significantly degrade downstream accuracy if not mitigated.
- AlpacaEval relies on GPT-4o-mini for scoring, which may introduce bias or inconsistency compared to human judges.
- Web search prompting is optional but significantly boosts performance; omitting it changes the evaluation conditions.
Evidence (verbatim from paper)
We evaluated AURA’s reasoning and tool use capabilities in two VoiceBench QA tasks: AlpacaEval (open-ended, scored 1–5 by GPT-4o-mini) and OpenBookQA (multiple-choice with ground-truth answers). AURA achieves an accuracy of 92.75% on OpenBookQA—competitive with GPT-4o and outperforming all other open-weight systems.
Citation
@misc{maben2025aura,
title={AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks},
author={Maben et al. (2025)},
year={2025},
note={arXiv:2506.23049}
}
- arXiv: 2506.23049