Stylebench Eval

stylebench-eval

StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control — Zhao et al. (2026) (arXiv:2603.07599, 2026)

What this evaluates

Evaluates speech language models' ability to control four paralinguistic dimensions (emotion, speed, volume, pitch) across multi-turn dialogues. It probes whether models can follow gradational style-intensity instructions while preserving fixed semantic content and maintaining coherent intensity trajectories across turns.

Datasets

• StyleBench — total 86400; splits: test (-1)

Metrics

• dimension-specific metrics (primary) — range: other
  • Quantitative measures applied per paralinguistic dimension (emotion, speed, volume, pitch) to assess how closely generated speech matches the target intensity level specified in the prompt.
• human evaluation — range: percent
  • Human raters assess the controllability and naturalness of the generated speech across the four style dimensions and intensity trajectories.

Input / output format

Input: Three-turn conversational dialogue. Each turn contains a user prompt with gradational style-intensity instructions and a fixed semantic answer text. Semantic content remains identical across turns; only the style instruction changes.

Output: Synthesized speech audio corresponding to the answer text, modulated to match the requested paralinguistic style and intensity level specified in the prompt.

Scoring recipe

for dialogue in StyleBench:
    for turn in dialogue.turns:
        audio = model.generate(turn.prompt, turn.text)
        score = compute_dimension_metric(audio, target_intensity)
        scores.append(score)
final_score = aggregate(scores)
# Human evaluation: raters score controllability and naturalness on a scale

Common pitfalls

• Models may alter the semantic content of the answer instead of just modulating the acoustic style, violating the fixed-text protocol.
• Evaluators might treat style control as categorical rather than gradational/intensity-based, missing the benchmark's core trajectory design.
• Ignoring the multi-turn intensity trajectory (increasing vs. decreasing) when scoring, which is a key controllability measure.

Evidence (verbatim from paper)

Evaluating the conversational speaking style control ability of SLMs necessitates coherent multi-turn dialogue modeling alongside accurate stylistic speech generation. We therefore construct a multi-turn QA dataset designed for controlling style and intensity. In this section, we outline StyleBench, a bilingual three-turn dialogue dataset that enables the evaluation of SLMs on: (1) understanding natural language instructions with gradational style variations, and (2) preserving hierarchical intensity control across dialogue turns. ... human evaluation and dimension-specific metrics providing fine-grained insights into controllability.

Citation

@misc{zhao2026stylebench,
  title={StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.07599}
}

• arXiv: 2603.07599