Lst Speech Text Bench Eval

lst-speech-text-bench-eval

Latent Speech-Text Transformer — Lu et al. (arXiv:2510.06195, 2025)

What this evaluates

Evaluates narrative understanding, commonsense reasoning, and topic coherence in speech-text models by selecting the most plausible continuation from multiple candidates. The benchmark tests both speech-to-speech and text-to-text modes to assess cross-modal alignment and reasoning capabilities under compute constraints.

Datasets

• HellaSwag (sHellaSWAG) — total ?; splits: test (-1)
• StoryCloze — total ?; splits: test (-1)
• TopicStoryCloze — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of instances where the model correctly selects the most plausible continuation from the given multiple-choice candidates.

Input / output format

Input: Narrative context (speech or text) followed by multiple candidate endings (speech or text).

Output: Selected candidate index or continuation text.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / len(predictions)

Common pitfalls

• Speech prompts and responses are generated independently with Kokoro TTS and concatenated, requiring models to handle modality switching correctly.
• Evaluation covers both speech-to-speech and text-to-text modes, so results must be reported separately for each modality.
• Datasets are resynthesized for higher-quality speech, meaning raw speech benchmarks may yield different results.

Evidence (verbatim from paper)

We evaluate the model on three benchmarks, where each dataset provides a narrative context and candidate endings, and the model selects the most plausible continuation. Table 2: Evaluation datasets for story completion (MC = Multiple Choice).

Citation

@misc{lu2025latentspeechtexttransformer,
  title={Latent Speech-Text Transformer},
  author={Lu et al.},
  year={2025},
  note={arXiv:2510.06195}
}

• arXiv: 2510.06195