Slam Omni Eval

slam-omni-eval

SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training — Wenxi Chen et al. (2024) (arXiv:2412.15649, 2024)

What this evaluates

This evaluation protocol assesses end-to-end spoken dialogue models across three core capabilities: instruction understanding, logical reasoning, and open-ended oral conversation. It measures both the semantic quality of the generated responses and the acoustic fidelity of the synthesized speech.

Datasets

• Repeat — total 252; splits: test (252)
• Summary — total 118; splits: test (118)
• StoralEval — total 201; splits: test (201)
• TruthfulEval — total 470; splits: test (470)
• MLC — total 177; splits: test (177)
• AlpacaEval — total 199; splits: test (199)
• CommonEval — total 200; splits: test (200)
• WildchatEval — total 349; splits: test (349)

Metrics

• ChatGPT Score (primary) — range: percent
  • Whisper-large-v3 transcribes the model's speech output into text. GPT-4o mini then scores the transcription against the gold text based on accuracy, relevance, clarity, and completeness using a predefined prompt.
• UTMOS Score — range: other
  • The UTMOS model predicts a Mean Opinion Score (MOS) to measure overall speech quality and naturalness.
• ASR-WER — range: percent
  • Word Error Rate calculated between the Whisper-large-v3 transcription of the model's speech output and the corresponding gold text response.

Input / output format

Input: User speech instruction (zero-padded to 30 seconds, encoded via Whisper-small, downsampled by factor k=5) paired with dialogue history.

Output: Speech response generated via greedy search decoding with repetition penalty 1.2, evaluated in non-streaming mode.

Scoring recipe

# 1. Transcribe model output
pred_text = whisper_large_v3.transcribe(model_speech_output)
gold_text = instance.gold_response

# 2. ChatGPT Score (Content Quality)
chatgpt_score = gpt4o_mini.score(
    transcription=pred_text,
    reference=gold_text,
    criteria=["accuracy", "relevance", "clarity", "completeness"]
)

# 3. UTMOS Score (Speech Quality)
utmos_score = utmos_model.predict_mos(model_speech_output)

# 4. ASR-WER (Alignment)
wer_score = calculate_wer(pred_text, gold_text)

# Aggregate across datasets
overall_chatgpt = mean(chatgpt_scores)
overall_utmos = mean(utmos_scores)
overall_wer = mean(wer_scores)

Common pitfalls

• Evaluating text-only LLMs by feeding them Whisper-transcribed audio without accounting for ASR errors in the input pipeline.
• Comparing non-streaming decoding results directly against streaming baselines without normalizing latency or chunking constraints.
• Assuming the synthesized training data (via CosyVoice) matches the distribution of the held-out evaluation benchmarks, which use real or differently synthesized prompts.

Evidence (verbatim from paper)

The model's inference results on these tasks are evaluated using the following metrics: ChatGPT Score To assess the content quality of the model's responses, we use Whisper-large-v3 to transcribe the speech output into text, followed by evaluation using GPT-4o mini (OpenAI, 2024a). The model is prompted to score the transcription based on predefined criteria, including accuracy, relevance, clarity, and completeness, with detailed prompts provided in Appendix C. UTMOS Score To measure the overall speech quality, we use the UTMOS (Saeki et al., 2022) model to predict mean opinion scores (MOS). WER Score To evaluate the speech-text alignment, we calculate the word error rate (WER) between the speech transcription and the corresponding text response, referred to as ASR-WER.

Citation

@misc{chen2024slamomni,
  title={SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training},
  author={Wenxi Chen et al. (2024)},
  year={2024},
  note={arXiv:2412.15649}
}

• arXiv: 2412.15649