Librispeech Wer Eval

librispeech-wer-eval

Semantic-aware Speech to Text Transmission with Redundancy Removal — Tianxiao Han et al. (2022) (arXiv:2202.03211, 2022)

What this evaluates

Evaluates the ability of a semantic-aware speech-to-text transmission system to accurately reconstruct text from speech signals under noisy communication channels (AWGN and Rayleigh). It probes semantic feature extraction, redundancy removal, and robustness to channel noise.

Datasets

• Librispeech — total ?; splits: train (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed by comparing the predicted transcription G~ against the ground truth transcription G. Standard substitution, deletion, and insertion errors are normalized by the reference length and expressed as a percentage.
• sentence similarity — range: [0, 1]
  • A metric measuring the semantic alignment between the predicted and ground truth sentences, computed alongside WER to assess content preservation beyond exact word matching.
• transmission efficiency — range: other
  • Average number of transmitted symbols per sentence, calculated from the output of the channel encoder after redundancy removal. Lower values indicate higher efficiency.

Input / output format

Input: Raw speech signals (converted to spectrum sequences) and corresponding ground truth transcriptions. During inference, only speech signals are input to the semantic encoder.

Output: Reconstructed text transcription (G~) and associated metrics (WER, sentence similarity, symbol count).

Scoring recipe

def compute_metrics(predictions, ground_truths):
    wer_scores = []
    sim_scores = []
    for pred, gold in zip(predictions, ground_truths):
        wer_scores.append(calculate_wer(pred, gold))
        sim_scores.append(calculate_sentence_similarity(pred, gold))
    avg_wer = sum(wer_scores) / len(wer_scores)
    avg_sim = sum(sim_scores) / len(sim_scores)
    return avg_wer, avg_sim

Common pitfalls

• Performance heavily depends on SNR regime; baselines degrade significantly at low SNR, making high-SNR comparisons insufficient.
• WER values may differ from prior DeepSC-SR reports (e.g., 20% vs 40%) due to specific feature extraction settings (40-dim fbank with derivatives).
• SE-DeepSC benchmark neglects semantic errors during transmission, leading to misleading upper-bound comparisons.

Evidence (verbatim from paper)

Algorithm 1 Testing algorithm of the proposed method. 1: Input: Speech signals and transcriptions G from dataset, fading channel h, noise w. 2: Set fading channel h = Rayleigh or AWGN 3: for each SNR value do 4: Generate Gaussian noise W under the SNR value. 5: Generate spectrum sequences S from input speech signals. 6: Output L from S by the semantic encoder. 7: Output X from L' by the channel encoder. 8: Transmit X and receive Y via (1). 9: Output L~ from Y by the channel decoder. 10: Output G~ from L' by the semantic decoder. 11: end for 12: Output: compare G and G and compute WER scores and sentence similarity via 2

Citation

@misc{han2022semantic,
  title={Semantic-aware Speech to Text Transmission with Redundancy Removal},
  author={Tianxiao Han et al. (2022)},
  year={2022},
  note={arXiv:2202.03211}
}

• arXiv: 2202.03211