Fastlongspeech Eval

fastlongspeech-eval

FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing — Guo et al. (2025) (arXiv:2507.14815, 2025)

What this evaluates

This evaluation protocol assesses the ability of Large Speech-Language Models to process and understand both short and long-form audio inputs across multiple tasks. It specifically probes speech comprehension, spoken question answering, dialogue understanding, emotion recognition, automatic speech recognition, and long-speech information retrieval under varying compression ratios.

Datasets

• LongSpeech-Eval — total ?; splits: test (-1)
• speech_QA_iemocap (AIR-Bench) — total ?; splits: test (-1)
• LibriSQA — total ?; splits: test (-1)
• LibriTTS (OpenASQA) — total ?; splits: test (-1)
• speech_dialogue_QA_fisher (AIR-Bench) — total ?; splits: test (-1)
• MELD — total ?; splits: test (-1)
• LibriSpeech — total ?; splits: test-clean (-1), test-other (-1)
• GigaSpeech — total ?; splits: test (-1)
• SPIRAL-H — total ?; splits: test (-1)

Metrics

• LLM-based QA Score (primary) — range: [1, 5]
  • Responses are scored on a scale of 1 to 5 by Llama3.1-70B-Instruct based on the question and ground-truth answer using a fixed prompt template.
• Word Error Rate (WER) — range: [0, 1]
  • Standard ASR metric calculated as the sum of substitutions, deletions, and insertions divided by the number of words in the reference transcript.
• Accuracy (ACC) — range: [0, 1]
  • Proportion of correctly predicted emotion labels out of the total number of samples.

Input / output format

Input: Audio speech files (for long-speech, split into 30-second clips before encoding) paired with text prompts or spoken questions.

Output: Text responses or answers generated by the model corresponding to the input questions.

Scoring recipe

def score_qa(predictions, questions, gold_answers, judge_model):
    scores = []
    for pred, q, gold in zip(predictions, questions, gold_answers):
        prompt = f"Question: {q}\nGold: {gold}\nPred: {pred}\nScore 1-5:"
        scores.append(int(judge_model.generate(prompt)))
    return sum(scores) / len(scores)

def wer(predictions, golds):
    return sum(edit_distance(p, g) for p, g in zip(predictions, golds)) / len(golds)

def accuracy(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

Common pitfalls

• Smaller target length L corresponds to a higher compression ratio, which is counterintuitive to many readers.
• Long-speech inputs are explicitly split into 30-second clips before audio encoding, meaning models cannot process the full raw waveform in a single forward pass without this segmentation step.
• LLM-based scoring relies heavily on the specific prompt template and judge model (Llama3.1-70B-Instruct), making cross-paper comparisons difficult without identical templates.

Evidence (verbatim from paper)

To evaluate the performance, we employ various metrics tailored to each task. For the Spoken QA and Spoken Dialogue Understanding task, we use Llama3.1-70B-Instruct to score responses on a scale of 1 to 5, with the scoring template available in the Appendix [D]. For the ASR task, we use Word Error Rate (WER) to assess the accuracy of the generated transcripts. For Emotion Recognition task, we use the Accuracy (ACC) metric to evaluate the performance.

Citation

@misc{guo2025fastlongspeech,
  title={FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2507.14815}
}

• arXiv: 2507.14815