Speechllm Eval

speechllm-eval

SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings — Yoo et al. (2025) (arXiv:2509.04473, 2025)

What this evaluates

Evaluates a unified speech-language model's ability to perform end-to-end automatic speech recognition (ASR), named entity recognition (NER), and sentiment analysis (SA) on low-resource speech datasets. It tests parameter-efficient adapter-based alignment of speech encoder features to a language model, along with classifier regularization and LoRA fine-tuning.

Datasets

• LibriSpeech — total ?; splits: train (-1), dev-clean (-1), test-clean (-1), test-other (-1); HF librispeech_asr
• SLUE-VoxPopuli — total ?; splits: dev (-1), test (-1)
• SLUE-VoxCeleb — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate, calculated as the number of insertions, deletions, and substitutions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.
• F1 (primary) — range: percent
  • Harmonic mean of precision and recall for NER or sentiment classification tasks. Calculated as 2 * (precision * recall) / (precision + recall).
• Label F1 — range: percent
  • F1 score computed at the label level for NER, aggregating precision and recall across all entity types before calculating the harmonic mean.
• SLUE Score (primary) — range: percent
  • An aggregate metric defined as the average of the primary metrics for ASR (WER), NER (F1), and SA (F1) tasks to provide a holistic view of multi-task performance.

Input / output format

Input: Raw speech audio recordings.

Output: Text transcripts (for ASR), sequence of NER entity tags, and sentiment class labels (Positive, Negative, Neutral).

Scoring recipe

def compute_wer(hypothesis, reference):
    return (edit_distance(hypothesis, reference) / len(reference.split())) * 100

def compute_f1(predictions, references):
    tp = sum(p == r for p, r in zip(predictions, references))
    fp = sum(p != r and r == 'other' for p, r in zip(predictions, references))
    fn = sum(p != r and p == 'other' for p, r in zip(predictions, references))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) * 100

def compute_slue_score(wer, ner_f1, sa_f1):
    return (wer + ner_f1 + sa_f1) / 3

Common pitfalls

• Comparing WER against baselines that use large beam decoding (size 500) and out-of-domain TED language models, which artificially lowers baseline WER and makes direct comparison unfair.
• Evaluating on all five sentiment classes instead of the three specified (Positive, Negative, Neutral) used in the official SLUE benchmark evaluation.
• Ignoring the impact of classifier regularizers and LoRA fine-tuning when comparing parameter efficiency or performance gains across different training stages.

Evidence (verbatim from paper)

The optimal adapter is selected by evaluating each epoch's adapter on the librispeech dev-clean set and selecting the best performing one based on the WER score.

Citation

@misc{yoo2025speechllm,
  title={SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings},
  author={Yoo et al. (2025)},
  year={2025},
  note={arXiv:2509.04473}
}

• arXiv: 2509.04473