Batonvoice Eval

batonvoice-eval

BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs — Wang et al. (2025) (arXiv:2509.26514, 2025)

What this evaluates

Evaluates a controllable text-to-speech model's ability to generate intelligible speech and accurately convey specific emotional tones based on text instructions. It probes zero-shot cross-lingual generalization and instruction-following capabilities in speech synthesis.

Datasets

• Seed-TTS — total ?; splits: test (-1)
• Emotion dataset — total 500; splits: test (500)

Metrics

• Word Error Rate (WER) — range: percent
  • Calculated by transcribing the generated speech with a pre-trained ASR model (Whisper-large-v3) and comparing it to the reference text. Lower values indicate higher intelligibility.
• Emotion Classification Accuracy (primary) — range: [0, 1]
  • Percentage of synthesized utterances correctly classified into one of five target emotions (joy, sadness, anger, surprise, fear) by an LLM judge (Gemini-2.5-Pro). Higher values indicate better emotional control.

Input / output format

Input: Text instructions (emotion labels or free-form descriptive prompts) and, for Seed-TTS, a short reference speech prompt.

Output: Synthesized speech audio (discrete vocal tokens or waveform) conditioned on the input text and instructed emotional style.

Scoring recipe

# Word Error Rate (WER)
asr_transcript = whisper_large_v3.transcribe(generated_audio)
wer = wer_score(reference_text, asr_transcript)

# Emotion Classification Accuracy
prompt = f"Classify the emotion of this speech: {generated_audio}"
pred_emotion = gemini_2_5_pro.classify(prompt)
accuracy = sum(pred_emotion == gold_emotion) / len(gold_emotion)

Common pitfalls

• WER scores depend heavily on the ASR backbone used (here, Whisper-large-v3), preventing direct cross-paper comparison without re-evaluation.
• Emotion accuracy uses a single LLM-as-a-judge (Gemini-2.5-Pro) instead of human raters, risking systematic bias or inconsistency.
• The emotion benchmark only covers five basic emotions, limiting assessment of nuanced or blended emotional control.

Evidence (verbatim from paper)

We measure performance using Emotion Classification Accuracy. This metric is derived by employing Google’s Gemini-2.5-Pro to classify the emotion of the synthesized speech. A higher accuracy indicates a greater success rate in generating perceptually accurate emotional speech.

Citation

@misc{wang2025batonvoice,
  title={BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.26514}
}

• arXiv: 2509.26514