Dipco Wer Eval

dipco-wer-eval

DiPCo -- Dinner Party Corpus — Van Segbroeck et al. (2019) (arXiv:1909.13447, 2019)

What this evaluates

Evaluates distant speech recognition and noise robustness by measuring word error rate on close-talk and far-field recordings of natural dinner conversations. It tests the model's ability to handle uncontrolled acoustic conditions, background music, and spatially diverse microphone placements.

Datasets

• DiPCo — total ?; splits: dev (-1), eval (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate, calculated as the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.

Input / output format

Input: Audio recordings (close-talk or far-field from a 7-microphone array) paired with reference transcripts.

Output: Predicted word sequence or transcript for each audio utterance.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Standard WER: min edit distance (ins, del, sub)
        dist = levenshtein(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Adaptation strategy must differ by recording type: close-talk recordings perform best without adaptation, while far-field recordings require adaptation (e.g., 4 iterations).
• The evaluation uses a leave-one-out cross-validation on the Dev set to tune meta-parameters, meaning the final Eval set results rely on a model adapted on all Dev sessions, not a single held-out fold.
• Rescoring with an interpolated 4-gram unpruned language model (LM4) yields the best WER, so baseline decoding without LM rescoring will underperform.

Evidence (verbatim from paper)

Baseline word error rate (WER) numbers were generated by using the Kaldi baseline (egs/chime5/s5) [2, 3] as the acoustic model (AM) source on which model adaptation was performed. The AM was a Time Delay Neural Network and Factored (TDNN-F) deep neural network [4] with 15 layers, the dimension of each layer was 1536 and the bottleneck dimension was 160. In a leave-one-out fashion, the Dev set was split by sessions to create five adaptation sets, each containing four sessions and with the fifth session used for verification purposes. This leave-one-out cross-validation process was set up to robustly infer the following meta-parameters: language model interpolation weight, number of iterations to use for adaptation, language model weight and word-insertion penalty. The adaptation was implemented as training with reduced learning rate (factor 10x smaller than the original learning rate). Note that we did not experiment with freezing only some of the layers, hence all network layers were retrained. After adaptation, the verification session was decoded.

Table 5 shows the results on the Eval set using the best set of meta parameters for a model that has been adapted using all Dev sessi

Citation

@misc{vansegbroeck2019dipco,
  title={DiPCo -- Dinner Party Corpus},
  author={Van Segbroeck et al. (2019)},
  year={2019},
  note={arXiv:1909.13447}
}

• arXiv: 1909.13447