Chime4 Ami Asr Eval

chime4-ami-asr-eval

Multi-Span Acoustic Modelling using Raw Waveform Signals — Patrick von Platen, Chao Zhang, Philip Woodland (arXiv:1906.11047, 2019)

What this evaluates

Evaluates automatic speech recognition systems on real-world meeting and close-talking microphone speech. It measures how well acoustic models trained on raw waveforms generalize to challenging multi-microphone environments compared to traditional feature-based baselines.

Datasets

• CHiME4 — total ?; splits: train (-1), dev (-1)
• AMI — total ?; splits: train (-1), dev (-1), eval (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.

Input / output format

Input: Raw waveform audio signals, shifted by 10ms (160 samples) per frame. Input is normalized to zero mean and unit variance (globally for CHiME4, per utterance and per meeting for AMI).

Output: Softmax probabilities over clustered triphone-states, decoded via Viterbi algorithm into word sequences using a language model.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Microphone failure detection algorithm must be applied to select the correct channel for the CHiME4 dev set.
• Normalization differs between datasets: CHiME4 uses global normalization, while AMI uses per-utterance zero mean and per-meeting unit variance.
• Alignment is done at 10ms intervals to clustered triphone-states, not directly to words.

Evidence (verbatim from paper)

Experimental results on CHiME4 and AMI datasets show a 5% relative WER reduction over FBANK-based models, with evidence that smaller kernels (50 params) outperform larger ones (400 params), and that learned filters diverge significantly from traditional log-Mel filters, indicating effective feature learning beyond handcrafted designs.

Citation

@misc{vonplaten2019multi,
  title={Multi-Span Acoustic Modelling using Raw Waveform Signals},
  author={Patrick von Platen, Chao Zhang, Philip Woodland},
  year={2019},
  note={arXiv:1906.11047}
}

• arXiv: 1906.11047