Spoken Mqa Eval

spoken-mqa-eval

Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems — Wei et al. (2025) (arXiv:2505.15000, 2025)

What this evaluates

Evaluates the ability of speech-based models to perform mathematical reasoning directly from audio inputs. It probes three distinct capabilities: direct arithmetic computation, contextual word-problem solving across single- and multi-step scenarios, and application of domain-specific mathematical knowledge in spoken language.

Datasets

• Spoken-MQA — total 2769; splits: arithmetic (273), contextual_reasoning_single_step (594), contextual_reasoning_multi_step (1402), knowledge_oriented_reasoning (500)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of questions where the model's predicted answer exactly matches the ground truth answer string. Standard exact-match evaluation for mathematical QA benchmarks.

Input / output format

Input: Audio recordings of mathematical problems spoken in natural language.

Output: Textual final answer (numerical value or short phrase).

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if str(pred).strip().lower() == str(gold).strip().lower():
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Verbal ambiguity in spoken math expressions (e.g., 'nine factorial' vs 'nine exclamation mark') can cause misinterpretation without visual context.
• TTS synthesis artifacts or pronunciation errors may degrade performance independently of the model's actual reasoning capability.
• Cascade ASR-to-LLM pipelines often lose intermediate reasoning steps, leading to incorrect final answers even if the ASR transcription is accurate.

Evidence (verbatim from paper)

We introduce the Spoken-MQA benchmark, a curated collection of diverse textual math problems designed to evaluate spoken mathematical reasoning. As illustrated in Figure[1], the benchmark is structured into three categories based on the type of mathematical reasoning required: (1) Arithmetic, which emphasizes direct numerical computation; (2) Contextual Reasoning, involving single- or multi-step reasoning grounded in real-world scenarios; and (3) Knowledge-Oriented Reasoning, which requires the application of mathematical domain-specific knowledge.

Citation

@misc{wei2025spokenmqa,
  title={Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2505.15000}
}

• arXiv: 2505.15000