qhjqhj00 mtbi-speech-eval
Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving — Jingran Xie et al. (2025) (arXiv:2505.18644, 2025)
What this evaluates
Evaluates the automatic speech recognition accuracy and zero-shot generalization capabilities of Speech Large Language Models. It probes robustness across mathematical reasoning, speaker role inference, and prompt adaptation tasks.
Datasets
- LibriSpeech — total ?; splits: test-clean (-1)
- GSM8K — total ?; splits: test (-1)
- Generalization Test Set — total ?; splits: test (-1)
Metrics
WER(primary) — range: percent- Standard ASR metric: (Substitutions + Deletions + Insertions) / Total Reference Words. Lower is better.
Prompt Generalization Accuracy— range: percent- Percentage of correctly answered prompts after ASR transcription, measuring robustness to speech-text alignment.
GSM8K Accuracy— range: percent- Exact-match accuracy on grade-school math word problems, evaluated in zero-shot and 1-shot settings.
Speaker Role Accuracy— range: percent- Percentage of correctly identified speaker roles in conversational prompts.
Input / output format
Input: Speech audio input paired with text prompts or task instructions.
Output: Text response (transcription for ASR, numerical/text answer for GSM8K, categorical label for Speaker Role).
Scoring recipe
def compute_metrics(predictions, golds, task):
if task == 'ASR':
return compute_wer(predictions, golds)
elif task in ['GSM8K', 'Speaker Role', 'Prompt Generalization']:
correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
return correct / len(golds) * 100
Common pitfalls
- ASR SFT models tend to overfit to transcription tasks, prioritizing ASR accuracy over prompt following, which artificially inflates prompt generalization scores.
- Mathematical reasoning tasks are highly sensitive to ASR transcription errors; even minor speech recognition mistakes cascade into incorrect final answers, lowering overall accuracy.
Evidence (verbatim from paper)
For evaluation, we use the standard ASR evaluation on the LibriSpeech test-clean dataset with word error rate (WER) as the evaluation metric. Details of the generalization evaluation are in Section 3. During inference, LLM applies a temperature of 0.7 and top-p sampling with a 0.85 threshold, setting a maximum token length of 100 (200 for math tasks) to minimize hallucinations.
Citation
@misc{xie2025mtbi,
title={Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving},
author={Jingran Xie et al. (2025)},
year={2025},
note={arXiv:2505.18644}
}
- arXiv: 2505.18644