Audiocrag Eval

audiocrag-eval

Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage — Arora et al. (2025) (arXiv:2510.02044, 2025)

What this evaluates

Evaluates speech-in-speech-out dialogue systems on their ability to accurately answer spoken queries using external tools, measuring both answer correctness and system latency under streaming versus open-book settings.

Datasets

• AudioCRAG — total ?; splits: synthetic (-1), human (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered queries out of the total number of queries in the benchmark.
• first-token latency — range: seconds
  • Time in seconds from query submission to the generation of the first output token, reported at P50 and P90 percentiles.

Input / output format

Input: Audio recordings of spoken user queries (synthetic or human-spoken) from the AudioCRAG benchmark.

Output: Text responses or audio responses (speech-in-speech-out) generated by the model.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_latency(latency_values):
    sorted_vals = sorted(latency_values)
    p50 = sorted_vals[len(sorted_vals)//2]
    p90 = sorted_vals[int(len(sorted_vals)*0.9)]
    return {'P50': p50, 'P90': p90}

Common pitfalls

• Latency measurements for synthetic audio exclude end-point detection latency, which is mandatory in production systems and would increase reported times.
• Speech output accuracy is consistently lower than text output accuracy due to difficulties in generating uncommon entity nouns directly in audio.
• Post-training is performed exclusively on synthetic data, so human-spoken benchmark results reflect zero-shot generalization rather than direct training.

Evidence (verbatim from paper)

All models are assessed on both the AudioCRAG-Synthetic (Syn.) and AudioCRAG-Human (Hum.). In the Closed Book setting, where models rely solely on their internal knowledge without access to external tools (reference length = 0), all models achieve accuracy scores below 20%.

Citation

@misc{arora2025streamrag,
  title={Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage},
  author={Arora et al. (2025)},
  year={2025},
  note={arXiv:2510.02044}
}

• arXiv: 2510.02044