Speech Rep Eval

speech-rep-eval

Entropy-based Coarse and Compressed Semantic Speech Representation Learning — Zuo et al. (2025) (arXiv:2509.00503, 2025)

What this evaluates

Evaluates the quality of compressed semantic speech representations across automatic speech recognition, speech-to-text translation, and voice conversion tasks. It probes how well adaptive entropy-based token aggregation preserves linguistic and acoustic information under varying compression ratios.

Datasets

• LibriSpeech — total ?; splits: test-clean (-1)
• CVSS-C — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: normalized edit distance between predicted and reference word sequences, expressed as a percentage.
• CER — range: percent
  • Character Error Rate: normalized edit distance between predicted and reference character sequences, expressed as a percentage.
• BLEU — range: [0, 100]
  • Bilingual Evaluation Understudy: geometric mean of modified n-gram precisions with a brevity penalty to penalize overly short translations.
• Q-MOS — range: [1, 5]
  • Quality Mean Opinion Score: subjective rating of naturalness and clarity on a 1-5 scale collected via human annotators.
• S-MOS — range: [1, 5]
  • Speaker Similarity Mean Opinion Score: subjective rating of speaker identity preservation on a 1-5 scale collected via human annotators.
• UTMOS — range: other
  • Automatic Mean Opinion Score prediction using a pretrained UTMOS model to estimate speech quality without human annotation.

Input / output format

Input: Raw speech audio or extracted HuBERT semantic units (k=500 clusters), optionally compressed via entropy-based token aggregation at specified Hz rates.

Output: For ASR/ST: autoregressively generated text tokens. For VC: synthesized mel-spectrogram.

Scoring recipe

def compute_metrics(predictions, references, audio_inputs=None):
    wer = 100 * edit_distance(predictions.words, references.words) / max(len(references.words), 1)
    cer = 100 * edit_distance(predictions.chars, references.chars) / max(len(references.chars), 1)
    bleu = compute_bleu(references.text, predictions.text)
    q_mos, s_mos = human_mos_ratings(predictions.audio) if audio_inputs else None
    utmos = utmos_model.predict(predictions.audio)
    latency = avg_decode_time(audio_inputs, predictions)
    return {'WER': wer, 'CER': cer, 'BLEU': bleu, 'Q-MOS': q_mos, 'S-MOS': s_mos, 'UTMOS': utmos, 'Latency': latency}

Common pitfalls

• Inference latency is hardware-dependent (measured on a single NVIDIA V100), so cross-device comparisons require normalization.
• VC quality relies on both subjective human ratings (Q-MOS/S-MOS) and automatic UTMOS; omitting either gives an incomplete quality assessment.
• Token rate thresholds (7, 15, 24 Hz) directly impact compression; results must be reported alongside the exact entropy threshold used.

Evidence (verbatim from paper)

We evaluate ASR and voice conversion using both objective and subjective metrics. ASR performance is assessed by Word Error Rate (WER) and Character Error Rate (CER), while voice conversion quality is evaluated via Q-MOS (naturalness and clarity) and S-MOS (speaker similarity) through human ratings on Amazon Mechanical Turk, complemented by automatic predictions using UTMOS. For speech-to-text translation, we report BLEU scores computed between the generated and reference text.

Citation

@misc{zuo2025entropy,
  title={Entropy-based Coarse and Compressed Semantic Speech Representation Learning},
  author={Zuo et al. (2025)},
  year={2025},
  note={arXiv:2509.00503}
}

• arXiv: 2509.00503