Minisuperb Eval

minisuperb-eval

MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models — Wang et al. (2023) (arXiv:2305.19011, 2023)

What this evaluates

Evaluates self-supervised speech models by measuring computational efficiency (forward MACs) and downstream task performance using a lightweight, offline feature extraction protocol. It probes the trade-off between model complexity and representation quality across speech tasks while enabling rapid early-stage model screening.

Datasets

• MiniSUPERB — total ?; splits: test (32); repo https://github.com/Comet0322/MiniSUPERB

Metrics

• forward MACs (primary) — range: other
  • Estimates the total number of multiply-accumulate operations (MACs) for the forward pass of both the upstream and downstream models.

Input / output format

Input: Speech utterances (32 samples from LibriSpeech test-clean) fed into a frozen upstream self-supervised speech model for offline representation extraction.

Output: Layer-wise hidden representations from the upstream model, processed through layer normalization and a weighted sum with trainable weights, then passed to the downstream model. Forward MAC count estimated via Microsoft DeepSpeed.

Scoring recipe

# Estimate forward MACs for upstream and downstream models
upstream_macs = deepspeed.estimate_macs(upstream_model, input_utterances, batch_size=1)
downstream_macs = deepspeed.estimate_macs(downstream_model, extracted_representations)
total_macs = upstream_macs + downstream_macs
return total_macs

Common pitfalls

• Using batch sizes > 1 during offline feature extraction can introduce padding and normalization artifacts that distort MAC estimates.
• Separating upstream and downstream computation improves training efficiency but may not reflect real-world inference latency where both run jointly.
• Evaluating only on 32 utterances provides a proxy for computational cost but may not capture full dataset performance variance.

Evidence (verbatim from paper)

To evaluate computational costs, we follow the setting in SUPERB Challenge and rely on Microsoft DeepSpeed333https://github.com/microsoft/DeepSpeed to estimate the forward MACs of both the upstream and downstream models. Following SUPERB Challenge, we perform this evaluation over the same 32 utterances sampled from the LibriSpeech test-clean dataset as inputs.

Citation

@misc{wang2023minisuperb,
  title={MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2305.19011}
}

• arXiv: 2305.19011