S2s Arena Eval

s2s-arena-eval

S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information — Feng Jiang et al. (2025) (arXiv:2503.05085, 2025)

What this evaluates

Evaluates speech-to-speech models on instruction following, assessing both semantic correctness and paralinguistic/speech quality in a reference-free, head-to-head comparison.

Datasets

• S2S-Arena — total ?; splits: test (-1); repo https://github.com/FreedomIntelligence/S2S-Arena

Metrics

• ELO score (primary) — range: other
  • Standard ELO rating system for pairwise comparisons. Ratings update after each match based on expected vs. actual outcome: R_new = R_old + K * (S - E), where S is actual score (1 for win, 0 for loss) and E is expected win probability derived from the rating difference.
• Pairwise win rate — range: percent
  • Percentage of pairwise comparisons won by a model against another specific model.

Input / output format

Input: Speech audio containing an instruction or query.

Output: Speech audio response generated by the model.

Scoring recipe

ratings = {model: 1000 for model in models}
for pair in comparisons:
    winner, loser = pair.winner, pair.loser
    expected = 1 / (1 + 10**((ratings[loser] - ratings[winner]) / 400))
    ratings[winner] += 32 * (1 - expected)
    ratings[loser] += 32 * (0 - expected)
return ratings

Common pitfalls

• Relying on automatic text-based evaluation (LLM-as-judge) loses critical paralinguistic and speech quality information.
• Speech-based automatic evaluators are unreliable and biased; human arena-style comparison is required.
• ELO scores are relative to the specific pool of models and evaluators, not absolute performance metrics.

Evidence (verbatim from paper)

Therefore, we adopt a manual arena-style approach with ELO ranking to more directly and comprehensively evaluate the performance of various speech models. More Details of ELO ranking calculation can be seen in the Appendix[C]. Followed by Chat-Arena, we build a S2S-Arena web-based evaluation tool for evaluators to perform a reference-free comparison. Given a speech as the input, we invite human evaluators to rank two speech outputs generated by different speech models, considering both semantics and speech quality, as shown in Figure[3].

Citation

@misc{jiang2025s2sarena,
  title={S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information},
  author={Feng Jiang et al. (2025)},
  year={2025},
  note={arXiv:2503.05085}
}

• arXiv: 2503.05085