qhjqhj00 instruct-tts-eval
InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems — Huang et al. (2025) (arXiv:2506.16381, 2025)
What this evaluates
Evaluates a text-to-speech system's ability to follow complex natural-language instructions for acoustic parameter specification, descriptive style direction, and role-play scenarios. It probes fine-grained prosodic control, open-ended style inference, and high-level scenario-based emotional/character expression.
Datasets
- InstructTTSEval — total ?; splits: EN-subset (-1), ZH-subset (-1); repo https://github.com/KexinHUANG19/InstructTTSEval
Metrics
accuracy(primary) — range: percent- Percentage of generated audio samples correctly matched to their corresponding natural-language instruction by the LLM judge (Gemini). Calculated as (number of correct matches / total evaluated samples) * 100.
Input / output format
Input: Natural-language instruction specifying acoustic parameters, style directives, or role-play scenarios.
Output: Synthesized audio file generated by the TTS model.
Scoring recipe
def compute_accuracy(predictions, instructions):
correct = 0
for audio, instruction in zip(predictions, instructions):
judge_output = gemini_judge(audio, instruction)
if judge_output == 'match':
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- LLM judges may exhibit self-preference bias, inflating scores for models from the same family (e.g., Gemini models scoring higher on Gemini-generated audio).
- Mismatched instruction-audio pairs might still partially align due to the many-to-many nature of speech and description, leading to false negatives in consistency checks.
- Safety filters in commercial models (e.g., Hume blocking child voices) can artificially lower scores unrelated to instruction-following capability.
Evidence (verbatim from paper)
Accuracy | APS | DSD | RP | Avg. | --- | --- | --- | --- | --- | | EN | 86% | 78% | 66% | 76.7% |
Citation
@misc{huang2025instructtts,
title={InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems},
author={Huang et al. (2025)},
year={2025},
note={arXiv:2506.16381}
}
- arXiv: 2506.16381