Step Audio R1 Eval

step-audio-r1-eval

Step-Audio-R1 Technical Report — Tian et al. (2025) (arXiv:2511.15848, 2025)

What this evaluates

Evaluates audio language models on speech understanding, reasoning, and real-time interactive dialogue capabilities using raw acoustic signals rather than textual transcriptions. It measures both comprehension accuracy across multiple audio benchmarks and real-time generation fluency.

Datasets

• Big Bench Audio — total ?; splits: test (-1); HF ArtificialAnalysis/big_bench_audio
• Spoken MQA — total ?; splits: test (-1)
• MMSU — total ?; splits: test (-1)
• MMAU — total ?; splits: test (-1)
• Wild Speech — total ?; splits: test (-1)

Metrics

• Average Score (%) (primary) — range: percent
  • Arithmetic mean of the percentage scores across Big Bench Audio, Spoken MQA, MMSU, MMAU, and Wild Speech benchmarks.
• Speech Reasoning Performance Score (%) — range: percent
  • Percentage score measuring the model’s reasoning ability over spoken content in the speech-to-speech benchmark.
• Latency (seconds) — range: seconds
  • Time in seconds to generate the first audio packet, quantifying response speed as an indicator of dialogue fluency.

Input / output format

Input: Raw audio recordings of spoken speech or acoustic signals.

Output: Text transcriptions or answers for speech-to-text tasks; spoken audio responses for speech-to-speech tasks.

Scoring recipe

# Speech-to-text evaluation
benchmarks = [BigBenchAudio, SpokenMQA, MMSU, MMAU, WildSpeech]
scores = [get_accuracy(model, bench) for bench in benchmarks]
avg_score = sum(scores) / len(scores)

# Speech-to-speech evaluation
reasoning_score = evaluate_reasoning(audio_input, audio_output)
latency = measure_first_packet_time(audio_output)

Common pitfalls

• Evaluating on transcribed text instead of raw acoustic signals, which defeats the purpose of modality-grounded reasoning.
• Comparing the standard Step-Audio-R1 model directly to the 'Realtime' variant's latency metrics without noting the architectural adaptation.
• Assuming the benchmarks are single-domain tests rather than a multi-task suite requiring an average score.

Evidence (verbatim from paper)

This benchmark comprises two evaluation metrics: the Speech Reasoning Performance Score (%), measuring the model’s reasoning ability over spoken content, and the first-packet Latency (seconds) metric, quantifying response speed as an indicator of dialogue fluency.

Citation

@misc{tian2025stepaudior1,
  title={Step-Audio-R1 Technical Report},
  author={Tian et al. (2025)},
  year={2025},
  note={arXiv:2511.15848}
}

• arXiv: 2511.15848