Superb Downstream Eval

superb-downstream-eval

An Experimental Study: Assessing the Combined Framework of WavLM and BEST-RQ for Text-to-Speech Synthesis — Nielson et al. (2023) (arXiv:2312.05415, 2023)

What this evaluates

Evaluates pre-trained speech models on downstream spoken language understanding tasks. It probes the model's ability to classify spoken intents, fill semantic slots in transcriptions, and detect specific keywords in audio.

Datasets

• SUPERB — total ?; splits: test (-1)

Metrics

• test accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted class labels out of the total number of instances. Calculated as correct predictions divided by total predictions.
• Slot_type_f1 — range: [0, 1]
  • The harmonic mean of precision and recall for identifying semantic slot types in spoken utterances, typically computed macro-averaged across all slot types.
• WER — range: percent
  • Word Error Rate, measuring the percentage of words incorrectly inserted, deleted, or substituted compared to the reference transcription.

Input / output format

Input: Raw audio recordings of spoken utterances.

Output: Predicted class labels for intent and keyword tasks, or token-level slot annotations and transcriptions for slot filling.

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1 = f1_score(golds, preds, average='macro')
    cer = edit_distance(golds, preds) / len(golds)
    wer = edit_distance(golds, preds) / len(golds)
    return {'accuracy': acc, 'f1': f1, 'cer': cer, 'wer': wer}

Common pitfalls

• Performance varies drastically between 100-hour and 960-hour training settings, making cross-experiment comparison difficult without normalization.
• The benchmark aggregates multiple distinct tasks with different metric directions (e.g., accuracy/F1 up vs. loss/CER/WER down), so reporting a single aggregate score can mask task-specific failures.
• CER and WER are highly sensitive to tokenization and normalization conventions, which may not be standardized across different SUPERB task implementations.

Evidence (verbatim from paper)

For the Keyword Spotting (KS) task, we observe a test accuracy of 25.41.

Citation

@misc{nielson2023experimental,
  title={An Experimental Study: Assessing the Combined Framework of WavLM and BEST-RQ for Text-to-Speech Synthesis},
  author={Nielson et al. (2023)},
  year={2023},
  note={arXiv:2312.05415}
}

• arXiv: 2312.05415