Flm Audio Eval

flm-audio-eval

FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Training — Yao et al. (2025) (arXiv:2509.02521, 2025)

What this evaluates

Evaluates native full-duplex audio-language models on speech understanding, speech generation, and real-time conversational capabilities. It measures how well models handle asynchronous text-audio streams, responsiveness to interruptions, and overall dialogue quality compared to specialized ASR/TTS systems and other full-duplex chatbots.

Datasets

• Fleurs-zh — total ?; splits: test (-1)
• LibriSpeech-clean — total ?; splits: test (-1)
• LlamaQuestions — total ?; splits: test (-1)
• Seed-TTS-en — total ?; splits: test (-1)
• Seed-TTS-zh — total ?; splits: test (-1)
• Custom Chinese Speech Instruction-Following Set — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: ratio of edit operations (insertions, deletions, substitutions) to the number of words in the reference transcript.
• Accuracy — range: [0, 1]
  • Exact-match accuracy for spoken question answering on LlamaQuestions.
• SIM — range: [0, 1]
  • Speaker Similarity score computed using a lightly fine-tuned speaker verification model on generated audio.
• LLM-score — range: [0, 10]
  • Quality score (0-10) assigned by DeepSeek-V3 comparing candidate textual responses to ground-truth answers.
• Human Evaluation Ratings — range: [0, 10]
  • Average ratings from 5 annotators across four dimensions: Helpfulness, Naturalness, Responsiveness, and Robustness.

Input / output format

Input: Audio prompts (converted from text for instruction-following), spoken questions, and reference audio clips for voice cloning/generation tasks.

Output: Text responses for ASR/QA/instruction-following tasks; generated audio waveforms for TTS evaluation.

Scoring recipe

def compute_metrics(predictions, references, generated_audio, reference_audio, candidate_text, ground_truth):
    wer = sum(edit_distance(p, r) / len(r) for p, r in zip(predictions, references)) / len(references)
    acc = sum(1 for p, r in zip(predictions, references) if p == r) / len(references)
    sim = speaker_similarity_model.score(generated_audio, reference_audio)
    llm_score = deepseek_v3.score(candidate_text, ground_truth)
    human_scores = [annotator.rates[dim] for annotator in annotators for dim in dimensions]
    return {'WER': wer, 'Accuracy': acc, 'SIM': sim, 'LLM-score': llm_score, 'Human': mean(human_scores)}

Common pitfalls

• Table 4 explicitly notes that different results for the same model come from different evaluation sources, potentially indicating different inference configurations.
• TTS similarity scores marked with (*) use a lightly fine-tuned similarity model, not the standard open-source speaker verification metric.
• The custom instruction-following benchmark converts text prompts to audio via a TTS pipeline before evaluation, so TTS quality indirectly affects instruction-following scores.

Evidence (verbatim from paper)

For ASR, we adopt word error rate (WER) as the primary metric, testing on both Chinese and English benchmarks, including Fleurs-zh [fleurs] and LibriSpeech-clean [librispeech]. While instruction-following with spoken input is addressed separately in Section[4.3], we also include LlamaQuestions [llamaquestions] as a speech-based QA benchmark, reporting accuracy.

Citation

@misc{yao2025flmaudio,
  title={FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Training},
  author={Yao et al. (2025)},
  year={2025},
  note={arXiv:2509.02521}
}

• arXiv: 2509.02521