Speech Drame Realism Eval

speech-drame-realism-eval

Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play — Jiatong Shi et al. (2025) (arXiv:2511.01261, 2025)

What this evaluates

Evaluates a speech foundation model's ability to generate role-play responses grounded in bottom-up, human-grounded realism. It measures fine-grained aspects such as prosodic dynamics, emotional fidelity, character consistency, and contextual fit based on realistic dialogue and media sources.

Datasets

• DRAME-RoleBench (Realism) — total ?; splits: test (-1); repo https://github.com/Anuttacon/speech_drame

Metrics

• realism_score (primary) — range: [0, 1]
  • A multi-dimensional scalar score assessing prosodic dynamics, emotional fidelity, character consistency, and contextual fit. Aggregated scores are correlated with human annotations using Pearson correlation to measure human alignment.

Input / output format

Input: Character profile ($\bm{C}{\text{profile}}$), scene specification ($\bm{C}{\text{scene}}$), and optionally preceding dialogue context ($\bm{C}_{\text{speech}}$).

Output: Generated speech response ($\bm{S}_r$), represented as a waveform or latent representation.

Scoring recipe

# For each instance in the benchmark:
response = model.generate(profile, scene, context)
score = sem.evaluate(response, profile, scene)
# Aggregate scores across the dataset
human_scores = get_human_annotations(instance)
pearson_corr = pearsonr(score, human_scores)
return pearson_corr

Common pitfalls

• Requires careful curation and more human annotation resources compared to archetype-based evaluation.
• Judgments are multi-dimensional, making it harder to isolate specific paralinguistic failures without detailed error analysis.
• Single-turn evaluation may not fully capture long-form conversational realism.

Evidence (verbatim from paper)

The second strategy adopts a bottom-up design, motivated by real-world performance and real human speech. Instead of relying solely on stereotypes, this approach grounds evaluation in recordings from professional and non-professional speakers, drawn from realistic dialogue and media sources. Judgments in this setting are more fine-grained and multi-dimensional, addressing aspects such as prosodic dynamics, emotional fidelity, character consistency, and contextual fit.

Citation

@misc{shi2025speechdrame,
  title={Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play},
  author={Jiatong Shi et al. (2025)},
  year={2025},
  note={arXiv:2511.01261}
}

• arXiv: 2511.01261