Slamming Additional Eval

slamming-additional-eval

Slamming: Training a Speech Language Model on One GPU in a Day — Maimon et al. (2025) (arXiv:2502.15814, 2025)

What this evaluates

Evaluates the speech language model's performance across multiple benchmarks including linguistic acceptability, story completion, audio generation quality, and cross-domain text generation perplexity.

Datasets

• sBLIMP — total ?; splits: test (-1)
• StoryCloze — total ?; splits: test (-1)
• People Speech — total ?; splits: test (-1)

Metrics

• sBLIMP — range: [0, 1]
  • Accuracy on linguistic acceptability judgments.
• tSC — range: [0, 1]
  • Accuracy on text story completion tasks.
• sSC — range: [0, 1]
  • Accuracy on speech story completion tasks.
• MOSnet (primary) — range: other
  • MOSnet score predicting mean opinion score for audio generation quality.
• GenPPL — range: other
  • Generation perplexity on out-of-domain prompts.
• BLEU — range: other
  • BLEU score for generated text quality.

Input / output format

Input: Text prompts for sBLIMP, StoryCloze, and People Speech; audio/text inputs for generation tasks.

Output: Generated text continuations or audio samples.

Scoring recipe

def compute_metrics(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    bleu = corpus_bleu([gold], predictions)
    ppl = exp(mean(cross_entropy_loss(predictions, gold)))
    mos = mosnet_model.predict(predictions)
    return {'accuracy': acc, 'bleu': bleu, 'ppl': ppl, 'mosnet': mos}

Common pitfalls

• MOSnet is a learned proxy metric, not a direct human MOS score.
• GenPPL and BLEU are evaluated on the People Speech test set, which may contain overlapping samples with baseline models' training data.
• Validation loss and ablation results are reported for specific context lengths and batch sizes rather than a single standardized setup.

Evidence (verbatim from paper)

For completeness we also provide MOS proxy results for our models compared to TWIST and Align-SLM models. We follow a similar setup to Lin et al. (2024) and use MOSnet to test the audio’s generation quality of our models. It is important to note that we use the same vocoder as TWIST and Align-SLM. The results can be seen in Table[10].

Citation

@misc{maimon2025slamming,
  title={Slamming: Training a Speech Language Model on One GPU in a Day},
  author={Maimon et al. (2025)},
  year={2025},
  note={arXiv:2502.15814}
}

• arXiv: 2502.15814