Cp Bench Eval

cp-bench-eval

Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Data — Wang et al. (2025) (arXiv:2509.16589, 2025)

What this evaluates

This benchmark evaluates speech-LLMs on contextual and paralinguistic reasoning tasks. It probes the models' ability to integrate linguistic content with emotional, prosodic, and social cues from in-the-wild speech data to answer specific question types.

Datasets

• CP-Bench — total ?; splits: test (-1)

Metrics

• LLaMA-3-70B judge score (primary) — range: [0, 100]
  • Responses are scored by LLaMA-3-70B-Instruct-AWQ on a 0–5 scale using a fixed evaluation prompt, then linearly rescaled to 0–100. Final scores are averaged over five independent judging runs per model to reduce variance.

Input / output format

Input: Audio clip (speech) paired with a natural language question targeting contextual, paralinguistic, or direct paralinguistic attributes.

Output: Free-form text response generated by the speech-LLM answering the question.

Scoring recipe

scores = []
for run in range(5):
    judge_input = f"Audio: {audio}, Question: {q}, Answer: {pred}"
    score_0_5 = llama_judge_score(judge_input)  # 0-5 scale
    score_0_100 = score_0_5 * 20
    scores.append(score_0_100)
return sum(scores) / len(scores)

Common pitfalls

• Qwen2-Audio has a strict 30-second input limit; the protocol requires evaluating the first and last 30 seconds separately and reporting the higher score.
• Azure OpenAI (GPT-4o) applies content filtering that blocks questions about speaker characteristics, artificially lowering its DP-category scores.
• Temperature defaults differ across models (0.7 for GPT, 1.0 for others), which significantly impacts performance and must be controlled for fair comparison.

Evidence (verbatim from paper)

Model outputs are evaluated using LLaMA-3-70B-Instruct-AWQ as a judge. The maximum context length for the loaded model is set to 4,096 tokens, encompassing both the input prompt and the generated output. The maximum number of new tokens generated by the judge per response is capped at 512. For generation, temperature and top_p were set to their default values of $1.0$ and $1.0$, respectively. Each response is scored on a scale from 0 to 5 using the evaluation prompt described in Section 3.3, and the scores are linearly rescaled to a 0–100 range for interpretability. To reduce variance, results are averaged over five independent runs of the judging process for each model’s predicted set, ensuring the reliability of our reported average scores.

Citation

@misc{wang2025cpbench,
  title={Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Data},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.16589}
}

• arXiv: 2509.16589