Ml Superb Eval

ml-superb-eval

Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction — Shi et al. (2023) (arXiv:2310.02720, 2023)

What this evaluates

Evaluates multilingual speech processing capabilities across 143 languages on ASR, Language Identification, and joint tasks under normal and few-shot settings. It probes cross-lingual transfer and low-resource adaptation of SSL models.

Datasets

• ML-SUPERB — total ?; splits: 10-minute (-1), 1-hour (-1); repo https://github.com/espnet/espnet/tree/master/egs2/ml_superb/asr1

Metrics

• ML-SUPERB score (primary) — range: other
  • Composite score defined by Shi et al. (2023a) aggregating performance across Monolingual ASR, Multilingual ASR, LID, and joint tasks. Task metrics include CER/PER and Accuracy.

Input / output format

Input: Frozen SSL representations fed into multilingual downstream architectures.

Output: Task-specific predictions (e.g., transcripts, language labels).

Scoring recipe

def compute_ml_superb_score(task_metrics):
    # Compute CER/PER for ASR tasks and Accuracy for LID
    # Aggregate into composite ML-SUPERB score per benchmark set (10-min/1-hour)
    ml_superb_score = aggregate_multilingual_metrics(task_metrics)
    return ml_superb_score

Common pitfalls

• Few-shot settings use very limited labeled data per language, making results highly sensitive to the downstream training recipe.
• The composite score masks per-language performance variations across the 143 languages.

Evidence (verbatim from paper)

We evaluate the performance of our proposed multilingual speech processing method using the ML-SUPERB benchmark... The ML-SUPERB benchmark comprises two sets of general benchmarks—specifically, a 10-minute set and a 1-hour set—across four tasks... we calculate a composite ML-SUPERB score as defined by Shi et al. (2023a)...

Citation

@misc{shi2023multiresolutionhubert,
  title={Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction},
  author={Shi et al. (2023)},
  year={2023},
  note={arXiv:2310.02720}
}

• arXiv: 2310.02720