Spatial Superb Eval

spatial-superb-eval

Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audio — Antoni Dimitriadis et al. (2023) (arXiv:2310.10922, 2023)

What this evaluates

Evaluates self-supervised speech representation models on downstream tasks including speaker identification, phoneme recognition, automatic speech recognition, emotion recognition, and speech localisation. It specifically probes robustness to noise and reverberation by comparing performance under clean versus noisy/reverberant training and testing conditions.

Datasets

• Spatial SUPERB — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• SID Accuracy — range: [0, 1]
  • Percentage of correctly identified speakers out of total samples.
• PR PER — range: percent
  • Phoneme error rate calculated as substitution, deletion, and insertion errors divided by total phonemes.
• ASR WER (primary) — range: percent
  • Word error rate calculated as (S+D+I)/N, where S, D, I are substitutions, deletions, insertions, and N is reference words.
• ER Accuracy — range: [0, 1]
  • Percentage of correctly classified emotion labels.
• Speech Localisation Angular Error — range: other
  • Angular distance in degrees between predicted and true direction of arrival (DOA).

Input / output format

Input: Multi-channel audio recordings (simulated first-order ambisonics) or single-channel audio, processed through a frozen upstream representation model to extract features. Downstream models are trained on these features using clean or noisy (0-20dB SNR) data.

Output: Task-specific predictions: speaker labels (SID), phoneme labels (PR), word transcriptions (ASR), emotion labels (ER), or continuous azimuth/elevation coordinates (Speech Localisation).

Scoring recipe

def compute_metrics(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    wer = edit_distance(predictions, gold) / len(gold)
    per = edit_distance(predictions, gold) / len(gold)
    ang_err = angular_distance(predictions, gold)
    return {'acc': acc, 'wer': wer, 'per': per, 'ang_err': ang_err}

Common pitfalls

• The Speech Localisation task uses discrete DOA labels with 11.25-degree quantization during training, which artificially limits achievable resolution and causes a baseline error of ~8 degrees even at high SNR.
• Performance heavily depends on whether the downstream model is trained on clean or noisy data; comparing clean-trained vs noisy-trained models without specifying the training condition leads to misleading conclusions.
• The paper reports upstream model performance via frozen downstream classifiers, so metric values reflect representation quality rather than end-to-end model tuning.

Evidence (verbatim from paper)

With language model decoding, Sp-HuBERT achieves greater than 40% reduction in WER when compared to WavLM Base+, along with significant improvements in SID.

Citation

@misc{dimitriadis2023spatialhubert,
  title={Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audio},
  author={Antoni Dimitriadis et al. (2023)},
  year={2023},
  note={arXiv:2310.10922}
}

• arXiv: 2310.10922