Chime3 Se Eval

chime3-se-eval

Cycle-Consistent Speech Enhancement — Meng et al. (2018) (arXiv:1809.02253, 2018)

What this evaluates

This benchmark evaluates speech enhancement models by measuring how well they clean noisy speech features before they are processed by a downstream automatic speech recognition (ASR) system. It probes the model's ability to preserve speech structure and reduce noise in challenging real-world far-field conditions.

Datasets

• CHiME-3 — total ?; splits: train (9137), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate (WER) computed by decoding enhanced speech features using a standard WSJ 5K word 3-gram language model and a clean DNN acoustic model.
• RWERR — range: percent
  • Relative WER Reduction calculated as (WER_baseline - WER_method) / WER_baseline * 100%.

Input / output format

Input: 87-dimensional vectors formed by appending 1st and 2nd order delta features to 29-dimensional log Mel filterbank (LFB) features, globally mean and variance normalized.

Output: 29-dimensional enhanced log Mel filterbank (LFB) features.

Scoring recipe

enhanced_features = model_F(noisy_features)
transcript = asr_decoder(
    enhanced_features, 
    lm="WSJ_5K_3gram", 
    am="clean_DNN"
)
wer = compute_wer(transcript, reference_transcript)
rwerr = (wer_baseline - wer) / wer_baseline * 100

Common pitfalls

• The evaluation metric is downstream ASR Word Error Rate (WER), not direct speech quality metrics like PESQ or STOI.
• The test set specifically uses only the 5th microphone channel from the CHiME-3 development set, not the full dataset.
• In Section 4.3, the acoustic model is re-trained on enhanced training data, which changes the evaluation protocol compared to the fixed acoustic model in Sections 4.1 and 4.2.

Evidence (verbatim from paper)

The real far-field noisy speech from the 5th microphone channel in CHiME-3 development data set is used for testing. We pre-train a clean DNN acoustic model as in Section 3.2 of [[31]] using 9137 clean training utterances in CHiME-3 to evaluate the ASR word error rate (WER) performance of the test features enhanced by F. The acoustic model is further re-trained with enhanced feature for better WERs. A standard WSJ 5K word 3-gram language model is used for decoding.

Citation

@misc{meng2018cycleconsistent,
  title={Cycle-Consistent Speech Enhancement},
  author={Meng et al. (2018)},
  year={2018},
  note={arXiv:1809.02253}
}

• arXiv: 1809.02253