Instructtts Eval

instructtts-eval

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions — Huang et al. (2026) (arXiv:2603.28086, 2026)

What this evaluates

Evaluates a text-to-speech model's capability to generate realistic vocal timbres that accurately follow complex natural-language style instructions. It probes fine-grained acoustic control, generalization to unstructured descriptions, and contextual role-play inference.

Datasets

• InstructTTSEval — total 6000; splits: test (6000)

Metrics

• Instruction-following accuracy (%) (primary) — range: percent
  • Percentage of generated audio clips that correctly satisfy the specified acoustic attributes, descriptive styles, or role-play scenarios in the instruction. Computed separately for each task (APS, DSD, RP) and language (EN, ZH).
• Pairwise preference win rate — range: percent
  • Percentage of comparisons where the model's output is preferred over a baseline's output by human annotators, calculated as (wins / total comparisons) * 100 across dimensions like Overall Performance, Instruction Following, and Naturalness.

Input / output format

Input: Natural language instruction describing vocal style/attributes + target text to synthesize.

Output: Generated audio clip.

Scoring recipe

# Objective: Instruction-following accuracy
correct = sum(1 for pred, gold in zip(predictions, golds) if pred.matches_instruction(gold))
accuracy = (correct / len(golds)) * 100

# Subjective: Pairwise preference win rate
wins = sum(1 for w, t, l in pairwise_results if w)
win_rate = (wins / len(pairwise_results)) * 100

Common pitfalls

• Test-set contamination must be prevented via fuzzy matching on transcripts, as training data overlaps with benchmark sources.
• Comparisons are invalid if baselines only support fixed voice editing rather than free-form instruction-driven design.
• Subjective evaluation uses majority vote from 3 annotators per item, with each annotator scoring only one dimension to reduce bias.

Evidence (verbatim from paper)

We evaluate MOSS-VoiceGenerator on InstructTTSEval, a public benchmark designed to assess TTS models’ ability to follow complex natural-language style instructions. InstructTTSEval comprises 6,000 test cases (3 tasks × 2 languages × 1,000 samples) drawn from movies, TV dramas, and variety shows, each paired with a reference audio clip. ... Table 1: Instruction-following accuracy (%) on InstructTTSEval.

Citation

@misc{huang2026mossvoicegenerator,
  title={MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2603.28086}
}

• arXiv: 2603.28086