Paras2s Eval

paras2s-eval

ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction — Shu-wen Yang et al. (2025) (arXiv:2511.08723, 2025)

What this evaluates

Evaluates how well speech-to-speech models adapt to and reflect paralinguistic styles (age, emotion, gender, sarcasm) in spoken responses. It measures both content appropriateness and stylistic alignment against ground-truth or human-annotated references.

Datasets

• ParaS2SBench — total ?; splits: test (-1)
• IEMOCAP — total ?; splits: test (-1)
• MELD — total ?; splits: test (-1)

Metrics

• ParaS2SBench score (primary) — range: [1, 5]
  • Average Likert-scale score (1–5) assigned to each prompt–response pair across four paralinguistic categories: Age, Emotion, Gender, and Sarcasm. Scores are generated by either human experts or an automatic GPT-based judge.

Input / output format

Input: Audio speech prompts containing specific paralinguistic styles (age, emotion, gender, sarcasm).

Output: Audio speech waveform representing the model's spoken response.

Scoring recipe

scores = []
for prompt, response in dataset:
    s_age = judge_score(prompt, response, "Age")
    s_emotion = judge_score(prompt, response, "Emotion")
    s_gender = judge_score(prompt, response, "Gender")
    s_sarcasm = judge_score(prompt, response, "Sarcasm")
    scores.append((s_age + s_emotion + s_gender + s_sarcasm) / 4)
return sum(scores) / len(scores)

Common pitfalls

• Models frequently produce 'tone-deaf' responses to contrasting styles, causing scores to average around 3.0 even when one response is highly appropriate.
• Automatic GPT-based scoring correlates strongly with human evaluation (>0.7), making human annotation unnecessary for routine benchmarking but critical only for validation.
• Comparing SFT and RL requires careful budget allocation; RL consistently outperforms SFT even when SFT uses 10x more data, so raw data volume comparisons are misleading.

Evidence (verbatim from paper)

The ParaS2SBench score is the average across 4 categories. ... Each prompt–response pair is scored by three human experts on a Likert scale... We also apply automatic scoring to study alignment.

Citation

@misc{yang2025paras2s,
  title={ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction},
  author={Shu-wen Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.08723}
}

• arXiv: 2511.08723