Meeting Asr Eval

meeting-asr-eval

GPU-accelerated Guided Source Separation for Meeting Transcription — Desh Raj et al. (2022) (arXiv:2212.05271, 2022)

What this evaluates

This evaluation protocol assesses the robustness of automatic speech recognition (ASR) systems in multi-talker meeting scenarios under varying microphone configurations (close-talk, distant, and source-separated). It measures transcription accuracy across different levels of speaker overlap and acoustic conditions, while also analyzing how diarization errors propagate to downstream ASR performance.

Datasets

• LibriCSS — total ?; splits: dev (-1), test (-1); repo https://github.com/desh2608/gss
• AMI — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/desh2608/gss
• AliMeeting — total ?; splits: train (-1), eval (-1), test (-1); repo https://github.com/desh2608/gss

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as (Insertions + Deletions + Substitutions) / Total Words. Reported as a percentage.
• CER — range: percent
  • Character Error Rate calculated as (Insertions + Deletions + Substitutions) / Total Characters. Used specifically for the AliMeeting dataset.
• DER — range: percent
  • Diarization Error Rate measuring the proportion of time incorrectly assigned to speakers or background. Computed without collars for LibriCSS and AMI, and with a 0.25s collar for AliMeeting.
• cpWER — range: percent
  • Concatenated minimum-permutation Word Error Rate, aligning predicted and reference speaker turns before computing WER to isolate ASR performance from diarization errors.

Input / output format

Input: Multi-channel audio recordings (individual headset, single distant, or GSS-enhanced multi-mic) organized into 10-minute mini-sessions or full meeting segments.

Output: Word-level or character-level transcriptions per audio segment, aligned with ground-truth speaker turns for cpWER/DER computation.

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref) for ref in references)
    if total_words == 0: return 0.0
    errors = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred, ref)
        errors += dist
    return (errors / total_words) * 100

Common pitfalls

• LibriCSS IHM is not real headset audio; it is simulated by concatenating LibriSpeech utterances, which may overestimate close-talk performance.
• AliMeeting uses CER instead of WER, and applies a 0.25s collar for DER computation, unlike LibriCSS and AMI which use no collar.
• GSS microphone setting uses different channel counts per dataset (7 channels for LibriCSS vs. 8 for AMI and AliMeeting), so direct cross-dataset comparison of channel counts is invalid.

Evidence (verbatim from paper)

For these experiments, we report diarization error rates (DER) and concatenated minimum-permutation WER (cpWER) [21] in order to analyze the impact of diarization errors on downstream ASR. We did not use any collars to compute DERs for LibriCSS and AMI, but a collar of 0.25 was used for AliMeeting following the original work.

Citation

@misc{raj2022gss,
  title={GPU-accelerated Guided Source Separation for Meeting Transcription},
  author={Desh Raj et al. (2022)},
  year={2022},
  note={arXiv:2212.05271}
}

• arXiv: 2212.05271