Superb Eval

superb-eval

Noise robust distillation of self-supervised speech models via correlation metrics — Ritter-Gutierrez et al. (2023) (arXiv:2312.12153, 2023)

What this evaluates

Evaluates speech models on three downstream tasks: Intent Classification, Keyword Spotting, and Automatic Speech Recognition. It specifically probes noise robustness by comparing performance on clean speech versus speech corrupted with out-of-distribution CHiMe3 background noise.

Datasets

• SUPERB — total ?; splits: test (-1)

Metrics

• Accuracy (Acc%) (primary) — range: percent
  • Percentage of correctly predicted labels out of the total number of test instances.
• Word Error Rate (WER%) — range: percent
  • Percentage of word-level errors (substitutions, deletions, and insertions) relative to the reference transcript.

Input / output format

Input: Frozen representations from the last hidden layer of the student speech model, fed into task-specific downstream classifiers or decoders. Inputs are either clean audio or audio corrupted with CHiMe3 background noise.

Output: Task-specific predictions: intent labels for IC, keyword labels for KS, and word transcripts for ASR.

Scoring recipe

def compute_metrics(predictions, references):
    acc = sum(1 for p, r in zip(predictions, references) if p == r) / len(references) * 100
    wer = wer_score(references, predictions) * 100  # standard WER implementation
    return {'accuracy': acc, 'wer': wer}

Common pitfalls

• The noisy test set specifically uses CHiMe3 background noise, not the Musan/WHAM/Gaussian noise used during training.
• Student model parameters are frozen during downstream evaluation; only the last hidden layer representations are passed to the task-specific heads.
• The 'diff' column in results represents the performance drop (clean minus noisy), not an independent metric.

Evidence (verbatim from paper)

Table 1: Accuracy (Acc% ↑) results on IC and KS and Word Error Rate (WER% ↓) results for ASR task on the SUPERB Benchmark for the previous approach versus our proposed method... Clean refers to the unmodified SUPERB test set, while noisy refers to the distorted set with CHiMe3 OOD noise. For downstream performance under clean and noisy settings, results on IC, KS and ASR tasks are reported. We report only out-of-distribution (OOD) noise to assess generalization to unseen noise. The OOD noise consists of a noise perturbation of the original clean speech with CHiMe3 noise.

Citation

@misc{rittergutierrez2023correlationdistillation,
  title={Noise robust distillation of self-supervised speech models via correlation metrics},
  author={Ritter-Gutierrez et al. (2023)},
  year={2023},
  note={arXiv:2312.12153}
}

• arXiv: 2312.12153