Longspeech Eval

longspeech-eval

LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech — Fei Yang et al. (2026) (arXiv:2601.13539, 2026)

What this evaluates

Evaluates long-form speech processing capabilities across transcription, translation, summarization, and higher-level reasoning tasks. It probes models' ability to maintain semantic consistency, track temporal progression, and extract structured information from ~10-minute audio segments.

Datasets

• LongSpeech — total 100000; splits: test (-1)

Metrics

• WER (primary) — range: [0, 1]
  • Word Error Rate: the ratio of word-level edit errors (insertions, deletions, substitutions) to total words in the reference transcription. Lower values indicate better performance.
• BLEU-4 (primary) — range: percent
  • n-gram precision with brevity penalty, computed case-insensitively. Higher scores indicate better translation fluency and adequacy.
• ROUGE-1/2/L F1 — range: [0, 1]
  • Recall-oriented n-gram overlap metrics measuring unigram, bigram, and longest common subsequence F1 scores. Higher values indicate better content coverage and coherence.
• Numeric Accuracy (primary) — range: [0, 1]
  • Proportion of responses that exactly match the ground truth numeric value.
• Parsability Rate — range: [0, 1]
  • Fraction of inputs where the model successfully parses the query structure (e.g., identifies target entity and operation).
• Post-Parsing Precision — range: [0, 1]
  • Accuracy of answers among successfully parsed queries.
• Misunderstanding Rate — range: [0, 1]
  • Proportion of cases where the model misinterprets the question intent (e.g., confuses entities or operations).
• Strict Accuracy (primary) — range: [0, 1]
  • Percentage of predictions matching the coarse label exactly, or ratio of fully correct 'YES' judgments for temporal localization.
• Relaxed Accuracy — range: [0, 1]
  • Rate at which the predicted label shares the same broad category/polarity, or ratio of 'YES' or 'PARTIALLY' judgments.
• Detection Accuracy — range: [0, 1]
  • Accuracy of language detection predictions.
• Detection Errors — range: [0, 1]
  • Error rate for language detection predictions.

Input / output format

Input: Audio segment (~10 minutes) paired with a task-specific text prompt (e.g., transcription instruction, translation query, summarization request, or QA question).

Output: Text response corresponding to the task: raw transcription, translated text, summary, numeric value, structured extraction, emotion category label, or temporal localization coordinates.

Scoring recipe

def score(predictions, golds, task):
    if task == 'ASR':
        return wer(predictions, golds)  # edit_dist / len(golds)
    elif task == 'S2TT':
        return bleu(predictions, golds)  # case-insensitive BLEU-4
    elif task == 'Summarization':
        return rouge_f1(predictions, golds, n=1,2,L)
    elif task in ['Content Separation', 'Speaker Count']:
        parsable = parse_query(predictions)
        num_acc = exact_match(predictions, golds)
        post_prec = accuracy(predictions[golds], parsable)
        mis_rate = 1 - parsability_rate(predictions)
        return num_acc, parsable, post_prec, mis_rate
    elif task == 'Emotion':
        coarse_pred = map_to_coarse(predictions)
        strict = exact_match(coarse_pred, golds)
        relaxed = match_polarity(coarse_pred, golds)
        return strict, relaxed
    elif task == 'Temporal':
        judgments = gpt4_judge(predictions, golds)  # YES/NO/PARTIALLY
        strict = (judgments == 'YES').mean()
        relaxed = (judgments in ['YES', 'PARTIALLY']).mean()
        return strict, relaxed

Common pitfalls

• Models may correctly parse the query intent but fail to extract the precise answer (e.g., high parsability rate but low numeric accuracy).
• Models lacking native long-audio support may output placeholder tokens (e.g., '[music]') instead of actual transcriptions, requiring segmentation-based evaluation pipelines.
• Emotion analysis requires mapping fine-grained model outputs to 7 predefined coarse categories before computing strict/relaxed accuracy.

Evidence (verbatim from paper)

We evaluate speech recognition performance using: Word Error Rate (WER): the ratio of word-level edit errors (insertions, deletions, substitutions) to total words in the reference transcription. Lower values indicate better performance. Speech-to-Text Translation For end-to-end translation from speech to text, we use BLEU: n-gram precision with brevity penalty, computed case-insensitively as BLEU-4. Higher scores indicate better translation fluency and adequacy.

Citation

@misc{yang2026longspeech,
  title={LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech},
  author={Fei Yang et al. (2026)},
  year={2026},
  note={arXiv:2601.13539}
}

• arXiv: 2601.13539