Instructttseval Zh Eval

instructttseval-zh-eval

VoiceSculptor: Your Voice, Designed By You — Jingbin Hu et al. (2026) (arXiv:2601.10629, 2026)

What this evaluates

Evaluates a text-to-speech model's ability to follow natural-language instructions for voice design, specifically controlling acoustic parameters, descriptive styles, and role-play characteristics. It measures how accurately synthesized speech adheres to explicit semantic and stylistic requirements.

Datasets

• InstructTTSEval-Zh — total ?; splits: test (-1)

Metrics

• APS — range: percent
  • Acoustic-Parameter Specification score (%). Measures adherence to explicit acoustic attribute instructions (e.g., pitch, rate, age).
• DSD — range: percent
  • Descriptive-Style Directive score (%). Measures adherence to descriptive style instructions (e.g., emotion, tone).
• RP — range: percent
  • Role-Play score (%). Measures adherence to role-playing or character-specific voice instructions.
• AVG (primary) — range: percent
  • Average of APS, DSD, and RP scores (%). Serves as the headline metric for overall instruction-following performance.
• IMOS — range: other
  • Instruction-following Mean Opinion Score. A human subjective rating on a standardized MOS-style scale assessing how well synthesized speech adheres to semantic and stylistic instructions.

Input / output format

Input: Natural language instruction specifying voice attributes (e.g., pitch, emotion, style) and optionally a target text for synthesis.

Output: Synthesized audio waveform matching the instructed voice characteristics.

Scoring recipe

def score_automated(predictions, instructions):
    aps_scores, dsd_scores, rp_scores = [], [], []
    for pred, inst in zip(predictions, instructions):
        res = llm_evaluate(audio=pred, instruction=inst) # LLM (e.g., Gemini 2.5 Pro) scores adherence
        aps_scores.append(res['APS'])
        dsd_scores.append(res['DSD'])
        rp_scores.append(res['RP'])
    aps = mean(aps_scores)
    dsd = mean(dsd_scores)
    rp = mean(rp_scores)
    avg = mean([aps, dsd, rp])
    return aps, dsd, rp, avg

def score_human(predictions, instructions):
    ratings = []
    for pred, inst in zip(predictions, instructions):
        rating = human_listener_rate(audio=pred, instruction=inst) # Standardized MOS scale
        ratings.append(rating)
    imos = mean(ratings)
    return imos

Common pitfalls

• LLM evaluator may exhibit bias toward commercial models or specific instruction phrasings.
• Evaluation is conducted exclusively on Chinese data, limiting claims about cross-lingual generalization.
• Commercial baselines are evaluated via official APIs while open-source models use local inference, potentially introducing infrastructure or latency biases.

Evidence (verbatim from paper)

To this end, we employ InstructTTSEval-Zh, a Chinese instruction-based TTS evaluation benchmark designed to measure how well a model follows natural-language instructions in speech synthesis. The benchmark evaluates multiple aspects of instruction controllability, including Acoustic-Parameter Specification (APS), Descriptive-Style Directive (DSD), and Role-Play (RP). These metrics are computed by synthesizing speech from instruction prompts and assessing the generated audio using a unified evaluation protocol with a large language model as the evaluator.

Citation

@misc{hu2026voicesculptor,
  title={VoiceSculptor: Your Voice, Designed By You},
  author={Jingbin Hu et al. (2026)},
  year={2026},
  note={arXiv:2601.10629}
}

• arXiv: 2601.10629