Speech Drame Archetype Eval

speech-drame-archetype-eval

Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play — Jiatong Shi et al. (2025) (arXiv:2511.01261, 2025)

What this evaluates

Evaluates a speech foundation model's ability to generate role-play responses that align with top-down, stereotype-driven character archetypes within specific scene contexts. It measures how well the model captures broad, accessible scoring criteria and general impressions of character consistency without relying on fine-grained prosodic nuance.

Datasets

• DRAME-RoleBench (Archetype) — total ?; splits: test (-1); repo https://github.com/Anuttacon/speech_drame

Metrics

• archetype_score (primary) — range: [0, 1]
  • A scalar score reflecting alignment with stereotypical archetypes and scene contexts. Model scores are aggregated and compared to human annotations using Pearson correlation to measure human alignment.

Input / output format

Input: Character profile ($\bm{C}{\text{profile}}$), scene specification ($\bm{C}{\text{scene}}$), and optionally preceding dialogue context ($\bm{C}_{\text{speech}}$).

Output: Generated speech response ($\bm{S}_r$), represented as a waveform or latent representation.

Scoring recipe

# For each instance in the benchmark:
response = model.generate(profile, scene, context)
score = sem.evaluate(response, profile, scene)
# Aggregate scores across the dataset
human_scores = get_human_annotations(instance)
pearson_corr = pearsonr(score, human_scores)
return pearson_corr

Common pitfalls

• Relies on broad stereotypes which may limit fine-grained prosodic or emotional nuance capture.
• Single-turn formulation ignores multi-turn dialogue context aggregation, which may skew long-conversation evaluations.
• Human alignment correlation (Pearson) is reported as a secondary metric, not the direct generation score.

Evidence (verbatim from paper)

The first strategy follows a top-down design, inspired by prior text-based role-play benchmarks. Here, role-play is judged with respect to stereotypical archetypes (e.g., “firefighter” or “ER doctor”), using scene contexts (e.g., “comfort a child trapped in fire” or “ask for help in a serious operation”). This approach provides accessible, general-purpose scoring that is scalable to a wide range of scenarios, making it suitable for large-scale benchmarking.

Citation

@misc{shi2025speechdrame,
  title={Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play},
  author={Jiatong Shi et al. (2025)},
  year={2025},
  note={arXiv:2511.01261}
}

• arXiv: 2511.01261