Dynamic Superb Phase2 Eval

dynamic-superb-phase2-eval

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks — Huang et al. (2024) (arXiv:2411.05361, 2024)

What this evaluates

Evaluates instruction-based universal speech and audio models across 180 tasks spanning speech, music, and environmental audio. It probes capabilities like automatic speech recognition, emotion recognition, speaker verification, and audio classification using a unified instruction-following framework.

Datasets

• Dynamic-SUPERB Phase-2 — total ?; splits: test (-1); repo https://github.com/dynamic-superb/dynamic-superb

Metrics

• relative_score (primary) — range: percent
  • Domain-level score averaging task-level relative improvements against a Whisper+LLaMA baseline. Task metrics are first scaled by the N/A rate: higher-is-better metrics are multiplied by (1 - N/A_rate), lower-is-better metrics are divided by (1 - N/A_rate). Relative improvement is then averaged per domain.
• WER — range: percent
  • Word Error Rate, standard ASR metric counting substitutions, deletions, and insertions normalized by reference words.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted class labels out of total instances.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for slot-filling and other classification tasks.
• PER — range: percent
  • Phoneme Error Rate, analogous to WER but at the phoneme level.
• CER — range: percent
  • Character Error Rate for slot value generation tasks.
• DER — range: percent
  • Diarization Error Rate measuring speaker turn confusion and misclassification.

Input / output format

Input: Raw audio waveform paired with a natural language instruction/prompt specifying the task (e.g., transcribe, classify, detect emotion, generate text/audio).

Output: Task-dependent outputs: text transcriptions, class labels, numerical values, or generated audio/text sequences. Models must strictly follow the instruction format to avoid high N/A rates.

Scoring recipe

def compute_domain_relative_score(tasks):
    domain_scores = []
    for task in tasks:
        raw_metric = evaluate_task(task.model_output, task.gold)
        baseline_metric = evaluate_task(task.baseline_output, task.gold)
        na_rate = count_na(task.model_output) / len(task.gold)
        
        if task.metric_direction == 'higher_is_better':
            scaled = raw_metric * (1 - na_rate)
            baseline_scaled = baseline_metric * (1 - na_rate)
        else:
            scaled = raw_metric / (1 - na_rate)
            baseline_scaled = baseline_metric / (1 - na_rate)
            
        relative_improvement = (scaled - baseline_scaled) / baseline_scaled
        domain_scores.append(relative_improvement)
    return sum(domain_scores) / len(domain_scores)

Common pitfalls

• Directly comparing Dynamic-SUPERB results to original SUPERB/HEAR benchmarks without accounting for the instruction-following adaptation and N/A rate scaling.
• Ignoring the N/A rate penalty for regression/generation tasks, which heavily penalizes models that fail to follow formatting instructions.
• Domain-level scores can be distorted by outliers with unbounded metrics (e.g., phoneme counting tasks with massive MSE).

Evidence (verbatim from paper)

For each task, we calculated the relative improvement of each model compared to the cascaded system baseline (Whisper + LLaMA) and then obtained the domain-level scores by averaging all its improvements across the tasks within each domain. For regression tasks, we introduced the N/A rate to measure how well a model meets the task requirements. The original task metric was only computed on instances that followed the task format (after post-processing by the LLM). To account for whether a model can follow instructions, we incorporated the N/A rate into the reported scores. Accordingly, we calculated the improvement based on scaled values. For metrics where a higher value indicates better performance (such as the F1 score), we multiplied the metric value by (1 – N/A rate). Conversely, for metrics where a lower value indicates better performance (such as word error rate), we divided the metric value by (1 – N/A rate).

Citation

@misc{huang2024dynamicsuperb,
  title={Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2411.05361}
}

• arXiv: 2411.05361