Lts Voiceagent Eval

lts-voiceagent-eval

LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning — Wenhao Zou et al. (arXiv:2601.19952, 2026)

What this evaluates

This evaluation probes the accuracy-latency-efficiency trade-off of streaming voice agents under realistic ASR conditions. It measures how well a system maintains reasoning quality while minimizing computational overhead and response delays when processing natural speech with disfluencies, misrecognitions, and non-uniform speaking rates.

Datasets

• VERA (AIME and GPQA-Diamond) — total ?; splits: test (-1)
• Spoken-MQA — total ?; splits: test (-1)
• BigBenchAudio — total ?; splits: test (-1)
• Pause-and-Repair Benchmark — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Standard exact-match or pass@1 accuracy on question-answering tasks, measuring the percentage of correctly answered queries.
• Time-to-First-Sentence (TTFS) — range: other
  • Wall-clock time from the start of audio input to the generation of the first complete sentence.
• Number of Forward-pass Evaluations (NFE) — range: other
  • Total count of LLM forward-pass executions performed during the inference process.
• Number of Interruption Times (NIT) — range: other
  • Count of mid-reasoning interruptions triggered by subsequent user speech or ASR revisions.
• Interruption Rate — range: percent
  • NIT divided by NFE, representing the fraction of forward passes that are interrupted by subsequent speech.

Input / output format

Input: Streaming audio processed by a unified internal ASR API at 200ms granularity (16kHz mono PCM, 3,200 samples per chunk), yielding incremental text transcripts fed to the LLM.

Output: Streaming text response generated via greedy decoding (temperature=0, max 4,096 new tokens) from the Qwen3-8B backbone.

Scoring recipe

def compute_metrics(predictions, golds, nfe, nit):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred.strip() == gold.strip())
    accuracy = (correct / len(golds)) * 100
    interruption_rate = (nit / nfe) * 100
    return {'accuracy': accuracy, 'interruption_rate': interruption_rate}
# TTFS and NFE are recorded directly from system logs during inference

Common pitfalls

• Evaluating the semantic trigger classifier in isolation rather than end-to-end, which fails to capture ASR jitter and the non-unique nature of spoken semantic boundaries.
• Comparing against baselines that use fixed text chunks or heuristic noise instead of realistic streaming ASR, leading to inflated efficiency claims.
• Confusing Time to First Token (TTFT) with Time-to-First-Sentence (TTFS), which better reflects perceived responsiveness in interactive voice agents.

Evidence (verbatim from paper)

We evaluate quality via Accuracy, latency via Time to First Token and Time-to-First-Sentence (TTFS), and efficiency via the Number of Forward-pass Evaluations (NFE) and the Number of Interruption Times (NIT). To quantify trigger validity, we report the Interruption Rate, defined as $ ext{NIT}/ ext{NFE}$; lower values indicate fewer mid-reasoning interruptions caused by subsequent speech and therefore more precise trigger timing.

Citation

@misc{zou2026ltsvoiceagent,
  title={LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning},
  author={Wenhao Zou et al.},
  year={2026},
  note={arXiv:2601.19952}
}

• arXiv: 2601.19952