Semascore

semascore

SeMaScore : a new evaluation metric for automatic speech recognition tasks — Zitha Sasindran et al. (2024) (arXiv:2401.07506, 2024)

What this evaluates

Evaluates automatic speech recognition (ASR) transcription quality by measuring segment-wise semantic similarity and error weighting. It specifically probes robustness on disordered, noisy, and accented speech, testing alignment with human judgments and downstream NLU task metrics.

Datasets

• Torgo — total 600; splits: test (600)
• Voicebank-DEMAND noisy speech testset — total ?; splits: test (-1)
• ATIS corpus (curated TTS) — total 500; splits: test (500)

Metrics

• SeMaScore (primary) — range: [0, 1]
  • Combines error-based metrics (e.g., MER) with semantic similarity computed via contextual embeddings (deberta-large-mnli), using Levenshtein-based sentence alignment to map hypothesis to reference segments.
• BERTScore — range: [0, 1]
  • Token-level cosine similarity between contextual embeddings of hypothesis and reference, aggregated across all token pairs.

Input / output format

Input: Audio speech utterance (or pre-transcribed hypothesis text) and corresponding ground truth reference text.

Output: A scalar similarity score per utterance or segment, plus optional alignment mappings.

Scoring recipe

def compute_semascore(hypothesis, reference, embedder='deberta-large-mnli'):
    # 1. Align hypothesis and reference segments using Levenshtein/edit distance
    aligned_pairs = levenshtein_align(hypothesis, reference)
    # 2. Compute semantic similarity for each aligned segment pair
    sims = [cosine_similarity(embedder(seg_h), embedder(seg_r)) for seg_h, seg_r in aligned_pairs]
    # 3. Calculate error rate (e.g., MER)
    err_rate = compute_mer(hypothesis, reference)
    # 4. Combine similarity and error weighting
    score = weighted_combine(sims, err_rate)
    return score

Common pitfalls

• BERTScore assigns high similarity scores to hypotheses that are semantically different from the ground truth in noisy or disordered speech.
• SeMaScore requires Levenshtein-based segment alignment rather than token-level mapping, which fundamentally changes score aggregation.
• Human assessment categories (0-2) are used only for correlation analysis, not as a direct metric output.

Evidence (verbatim from paper)

All contextual embeddings were obtained using pre-trained deberta-large-mnli model. Revisiting the examples in Table 1 where BERTscore fails, we can see that SeMaScore evaluated the hypotheses better by penalizing them appropriately.

Citation

@misc{zitha2024semascore,
  title={SeMaScore : a new evaluation metric for automatic speech recognition tasks},
  author={Zitha Sasindran et al. (2024)},
  year={2024},
  note={arXiv:2401.07506}
}

• arXiv: 2401.07506