Speechparaling Bench Eval

speechparaling-bench-eval

SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation — Liu et al. (2026) (arXiv:2604.20842, 2026)

What this evaluates

Evaluates large audio-language models (LALMs) on their ability to generate speech with fine-grained paralinguistic features, including dynamic intra-utterance variation and context-aware adaptation. It probes how well models interpret and modulate tone, pitch, emotion, and non-linguistic vocalizations in response to textual instructions and contextual cues.

Datasets

• SpeechParaling-Bench — total 1000; splits: test (1000)

Metrics

• Judge Score (0-100) (primary) — range: [0, 100]
  • Pairwise preference evaluation using an LALM-based judge. The judge compares model outputs and assigns scores normalized to a 0–100 scale. The reported metric is the average score across all evaluated samples.
• Spearman's rank correlation coefficient — range: [-1, 1]
  • Non-parametric measure of rank correlation between the rankings produced by the automated judge and those produced by human evaluators on a 5% random subset (416 pairs).

Input / output format

Input: Text prompt specifying the desired paralinguistic behavior, along with contextual audio/text cues (e.g., scenario description, pre-set tone, or user utterance).

Output: Generated audio response matching the specified paralinguistic and contextual constraints.

Scoring recipe

def evaluate(dataset, model):
    pairs = generate_response_pairs(dataset, model)
    scores = []
    for pair in pairs:
        raw_score = llm_judge.evaluate(pair.audio_a, pair.audio_b, prompt=evaluation_criteria)
        scores.append(normalize_to_0_100(raw_score))
    return mean(scores)

def human_agreement(model_rankings, human_rankings):
    return spearmanr(model_rankings, human_rankings).correlation

Common pitfalls

• Models often default to generic, formalized responses due to 'voice assistant' identity constraints, failing to adapt to complex emotional or role-play contexts.
• Dynamic intra-utterance variation is frequently missed because paralinguistic features are strongly coupled with linguistic content and training data lacks explicit variation examples.
• Sarcasm and subtle non-linguistic vocalizations (e.g., laughter) are frequently overlooked, leading to literal interpretations and inappropriate tone matching.

Evidence (verbatim from paper)

Following the previous approach, we conduct a human evaluation to test how well model-based judgment aligns with human subjective preference. Human judges are instructed to determine the winner (or tie) of each response pair, with instructions and evaluation criteria similar to those in the model-judging prompt. We randomly select 5% samples from each task, totaling 416 response audio pairs. We compute Spearman’s rank correlation coefficient to measure the agreement between rankings derived from model judgments and those from human evaluators. The average of the initial scores of all evaluated models given by the judge model (scores normalized to 0–100) is reported as the main metric.

Citation

@misc{liu2026speechparalingbench,
  title={SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.20842}
}

• arXiv: 2604.20842