# Speechr Eval

> Probes speech reasoning capabilities in large audio-language models across factual, procedural, and normative dimensions. It tests whether models can perform multi-step inference, maintain logical coherence, and make normative judgments when processing spoken input under varying prosodic and emotional conditions. Use when the user wants to benchmark on SpeechR, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/speechr-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/speechr-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/speechr-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/speechr-eval

---


# speechr-eval

> SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models — Wanqi Yang et al. (2025) (arXiv:2508.02018, 2025)

## What this evaluates

Probes speech reasoning capabilities in large audio-language models across factual, procedural, and normative dimensions. It tests whether models can perform multi-step inference, maintain logical coherence, and make normative judgments when processing spoken input under varying prosodic and emotional conditions.

## Datasets

- **SpeechR** — total ?; splits: test (-1); repo https://github.com/Yanda95/SpeechR

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions among all evaluated instances.
- `Final Correctness` — range: [0, 1]
  - Binary score (0 or 1) indicating whether the generated answer matches the reference answer.
- `Logical Relevance` — range: [1, 5]
  - Integer score (1 to 5) indicating whether the answer logically follows from the question.
- `CoT Coherence` — range: [1, 5]
  - Integer score (1 to 5) indicating whether the reasoning chain is internally consistent and well-structured.

## Input / output format

**Input**: Audio recording of a spoken question or statement, accompanied by a standardized prompt. Multiple-choice and acoustic-feature versions include answer options; the generative version prompts the model to produce a reasoning chain and final answer.

**Output**: Multiple-choice/Acoustic: Discrete option label (e.g., “A”, “B”) or binary decision. Generative: Open-ended text containing a step-by-step reasoning chain followed by a final answer.

## Scoring recipe

```python
if format in ['multiple_choice', 'acoustic_feature']:
    pred_label = extract_valid_option(prediction)
    correct = 1 if pred_label == ground_truth else 0
    metric_value = sum(correct) / len(dataset)  # Accuracy
elif format == 'generative':
    judge = GPT-4o
    fc = judge.evaluate(prediction, ground_truth, rubric='Final Correctness (0/1)')
    lr = judge.evaluate(prediction, question, rubric='Logical Relevance (1-5)')
    coh = judge.evaluate(prediction, rubric='CoT Coherence (1-5)')
    metric_value = {'Final Correctness': fc, 'Logical Relevance': lr, 'CoT Coherence': coh}
```

## Common pitfalls

- Assuming high automatic speech recognition (ASR) accuracy implies strong reasoning performance; the benchmark shows a critical disconnect between transcription proficiency and actual inference/coherence.
- Modifying or rephrasing model outputs before LLM-as-a-judge evaluation introduces bias; the protocol strictly requires passing raw outputs without post-processing.
- Evaluating only on clean audio ignores the benchmark's core design to test robustness against prosodic stress and emotional tone variations.

## Evidence (verbatim from paper)

> We use a discrete-choice evaluation, where model outputs are scanned for valid option labels (e.g., “A”, “B”) and matched to the ground-truth answer. Accuracy is the proportion of correct predictions. ... Final Correctness (0 or 1): Binary score indicating whether the answer matches the reference. • Logical Relevance (1 to 5, integer): Whether the answer logically follows from the question. • CoT Coherence (1 to 5, integer): Whether the reasoning is internally consistent and well-structured.

## Citation

```bibtex
@misc{yang2025speechr,
  title={SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models},
  author={Wanqi Yang et al. (2025)},
  year={2025},
  note={arXiv:2508.02018}
}
```

- arXiv: 2508.02018