Discourse Stress Tts Eval

discourse-stress-tts-eval

Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark — Turetzky et al. (2026) (arXiv:2604.10580, 2026)

What this evaluates

Evaluates whether text-to-speech systems can correctly realize discourse-dependent word-level stress based on contrasting contexts. It probes the model's ability to adapt prosodic emphasis dynamically rather than relying on fixed sentence-internal stress patterns.

Datasets

• CAST — total 10000; splits: test (10000)

Metrics

• Hit — range: [0, 1]
  • 1[A ∈ Ŷ_A] where A is the intended stressed word and Ŷ_A is the set of detected stressed words in the synthesized audio. Measures basic stress expressiveness regardless of additional stressed words.
• Pair-Contrast — range: [0, 1]
  • 1[A ∈ Ŷ_A ∧ B ∉ Ŷ_A] requiring the correct stressed word to be present while the alternative context's stressed word is absent. Penalizes over-expressive stress.
• Pair-Correct (primary) — range: [0, 1]
  • 1[(A ∈ Ŷ_A ∧ B ∉ Ŷ_A) ∧ (B ∈ Ŷ_B ∧ A ∉ Ŷ_B)] requiring Pair-Contrast to hold for both sides of a contrastive context pair simultaneously. The strictest metric, ensuring genuine context adaptation.

Input / output format

Input: Target sentence paired with a contrasting context, provided via conditioning modes (no context, prepended context, natural language instruction, or explicit stress markup).

Output: Synthesized audio waveform for the target sentence.

Scoring recipe

# predictions: detected stressed words per sample (Ŷ_A, Ŷ_B)
# gold: intended stressed words per sample (A, B)
hit = 1.0 if A in Ŷ_A else 0.0
pair_contrast = 1.0 if (A in Ŷ_A) and (B not in Ŷ_A) else 0.0
pair_correct = 1.0 if (pair_contrast and (B in Ŷ_B) and (A not in Ŷ_B)) else 0.0
return hit, pair_contrast, pair_correct

Common pitfalls

• Systems with a sentence-internal bias toward stressing a specific word may artificially satisfy Pair-Contrast for one side of a pair by chance.
• Over-expressive stress (stressing multiple words) will cause a system to fail Pair-Contrast even if the target word is correctly emphasized.
• The automatic stress detector (WHISTRESS) is used as a proxy for human perception, and its moderate agreement (κ=0.32) means metric scores may not perfectly align with human judgments.

Evidence (verbatim from paper)

Pair-Correct (Correct): a pair-level metric requiring Pair Contrast to hold for both sides of the contrastive context pair: $$ \mathbf {1} \left[ \left(A \in \hat {Y} _ {A} \wedge B \notin \hat {Y} _ {A}\right) \wedge \left(B \in \hat {Y} _ {B} \wedge A \notin \hat {Y} _ {B}\right)\right]. $$ This is the strictest metric, requiring Pair Contrast to hold for both sides simultaneously.

Citation

@misc{turetzky2026knowing,
  title={Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark},
  author={Turetzky et al. (2026)},
  year={2026},
  note={arXiv:2604.10580}
}

• arXiv: 2604.10580