Speechr Eval

speechr-eval

SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models — Wanqi Yang et al. (2025) (arXiv:2508.02018, 2025)

What this evaluates

Probes speech reasoning capabilities in large audio-language models across factual, procedural, and normative dimensions. It tests whether models can perform multi-step inference, maintain logical coherence, and make normative judgments when processing spoken input under varying prosodic and emotional conditions.

Datasets

• SpeechR — total ?; splits: test (-1); repo https://github.com/Yanda95/SpeechR

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions among all evaluated instances.
• Final Correctness — range: [0, 1]
  • Binary score (0 or 1) indicating whether the generated answer matches the reference answer.
• Logical Relevance — range: [1, 5]
  • Integer score (1 to 5) indicating whether the answer logically follows from the question.
• CoT Coherence — range: [1, 5]
  • Integer score (1 to 5) indicating whether the reasoning chain is internally consistent and well-structured.

Input / output format

Input: Audio recording of a spoken question or statement, accompanied by a standardized prompt. Multiple-choice and acoustic-feature versions include answer options; the generative version prompts the model to produce a reasoning chain and final answer.

Output: Multiple-choice/Acoustic: Discrete option label (e.g., “A”, “B”) or binary decision. Generative: Open-ended text containing a step-by-step reasoning chain followed by a final answer.

Scoring recipe

if format in ['multiple_choice', 'acoustic_feature']:
    pred_label = extract_valid_option(prediction)
    correct = 1 if pred_label == ground_truth else 0
    metric_value = sum(correct) / len(dataset)  # Accuracy
elif format == 'generative':
    judge = GPT-4o
    fc = judge.evaluate(prediction, ground_truth, rubric='Final Correctness (0/1)')
    lr = judge.evaluate(prediction, question, rubric='Logical Relevance (1-5)')
    coh = judge.evaluate(prediction, rubric='CoT Coherence (1-5)')
    metric_value = {'Final Correctness': fc, 'Logical Relevance': lr, 'CoT Coherence': coh}

Common pitfalls

• Assuming high automatic speech recognition (ASR) accuracy implies strong reasoning performance; the benchmark shows a critical disconnect between transcription proficiency and actual inference/coherence.
• Modifying or rephrasing model outputs before LLM-as-a-judge evaluation introduces bias; the protocol strictly requires passing raw outputs without post-processing.
• Evaluating only on clean audio ignores the benchmark's core design to test robustness against prosodic stress and emotional tone variations.

Evidence (verbatim from paper)

We use a discrete-choice evaluation, where model outputs are scanned for valid option labels (e.g., “A”, “B”) and matched to the ground-truth answer. Accuracy is the proportion of correct predictions. ... Final Correctness (0 or 1): Binary score indicating whether the answer matches the reference. • Logical Relevance (1 to 5, integer): Whether the answer logically follows from the question. • CoT Coherence (1 to 5, integer): Whether the reasoning is internally consistent and well-structured.

Citation

@misc{yang2025speechr,
  title={SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models},
  author={Wanqi Yang et al. (2025)},
  year={2025},
  note={arXiv:2508.02018}
}

• arXiv: 2508.02018