Voiceagentbench Eval

voiceagentbench-eval

VoiceAgentBench: Are Voice Assistants ready for agentic tasks? — Dhruv Jain et al. (2025) (arXiv:2510.07978, 2025)

What this evaluates

Evaluates speech language models and ASR-LLM pipelines on agentic speech tasks. It probes single/multi-tool orchestration, multi-turn dialogue, and safety refusal capabilities across multiple languages, including English, Hindi, and five Indic languages.

Datasets

• VoiceAgentBench — total 5500; splits: English (-1), Hindi (-1), Indic (-1), Source-native (-1), Safety (-1)

Metrics

• PF (Parameter Filling) (primary) — range: percent
  • Percentage of queries where the model correctly identifies the target tool, matches the required schema, and accurately fills all necessary parameters. Calculated as the average across task categories.
• TS (Tool Selection) — range: percent
  • Binary accuracy for correctly identifying the intended tool from the available set. Often near-perfect for single-tool tasks.
• TCS (Tool Call Schema) — range: percent
  • Binary accuracy for correctly structuring the tool call according to the required schema/format, independent of parameter values.
• Refusal Rate — range: percent
  • Percentage of safety/harmful queries where the model explicitly refuses to execute the request or generate a tool call.

Input / output format

Input: Synthetic spoken audio queries (or transcribed text for ASR-LLM baselines) accompanied by tool definitions, system instructions, and multi-turn dialogue context.

Output: Structured tool call arguments (tool name, schema, parameters) or a natural language refusal response.

Scoring recipe

def score_instance(pred, gold):
    ts = 1 if pred['tool_name'] == gold['tool_name'] else 0
    tcs = 1 if pred['schema'] == gold['schema'] else 0
    pf = 1 if (tcs == 1 and pred['params'] == gold['params']) else 0
    return {'TS': ts, 'TCS': tcs, 'PF': pf}

def compute_metrics(predictions, golds):
    scores = [score_instance(p, g) for p, g in zip(predictions, golds)]
    return {
        'TS': sum(s['TS'] for s in scores) / len(scores),
        'TCS': sum(s['TCS'] for s in scores) / len(scores),
        'PF': sum(s['PF'] for s in scores) / len(scores)
    }

Common pitfalls

• TS scores are trivially high for single-tool calling tasks and do not reflect true agentic capability; PF is the meaningful metric for tool execution.
• PF performance heavily depends on ASR transcription quality for Indic languages; replacing ground-truth transcripts can boost PF by >24%, masking model reasoning limits.
• Safety/refusal rates are highly sensitive to explicit refusal prompts in the system prompt; removing them causes sharp drops in refusal rates.

Evidence (verbatim from paper)

Metrics include TS, TCS, and PF (see Section [3.3] for definitions). For Indic subset we report average across all the 5 Indic languages. TS for Single Tool Calling is trivial, often yielding near-perfect scores.

Citation

@misc{jain2025voiceagentbench,
  title={VoiceAgentBench: Are Voice Assistants ready for agentic tasks?},
  author={Dhruv Jain et al. (2025)},
  year={2025},
  note={arXiv:2510.07978}
}

• arXiv: 2510.07978