Speechinstructbench Eval

speechinstructbench-eval

InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training — Wang et al. (2025) (arXiv:2503.02769, 2025)

What this evaluates

Evaluates speech instruction-following capabilities across closed-ended, open-ended, and adjustment tasks under varying acoustic conditions (background noise, accents, disfluencies) in English and Chinese.

Datasets

• SpeechInstructBench — total ?; splits: test (-1)

Metrics

• prompt-level accuracy (P) — range: percent
  • Computed by averaging loose and strict accuracies for closed-ended questions.
• instruction-level accuracy (I) (primary) — range: percent
  • Computed by averaging loose and strict accuracies for closed-ended questions.
• Instruction Adherence Rate (IAR) — range: percent
  • Measures the rate of correct instruction adherence for adjustment tasks; higher is better.
• Error Correction Rate (ECR) — range: percent
  • Measures the error correction rate for adjustment tasks; lower is better.

Input / output format

Input: 16kHz resampled speech segments converted to 128-channel mel-spectrograms (25ms window, 10ms hop, stride-2 pooling), provided alongside text prompts or instructions.

Output: Text response predicting the continuation or answering the instruction.

Scoring recipe

def compute_metrics(predictions, gold):
    loose_acc = match_loose(predictions, gold)
    strict_acc = match_strict(predictions, gold)
    p_acc = (loose_acc + strict_acc) / 2
    i_acc = (loose_acc + strict_acc) / 2
    iar = count_adherence(predictions, gold) / len(gold)
    ecr = count_errors(predictions, gold) / len(gold)
    return {'P': p_acc, 'I': i_acc, 'IAR': iar, 'ECR': ecr}

Common pitfalls

• P and I accuracies are averaged from loose and strict versions, not reported separately.
• ECR is a lower-is-better metric (↓) while IAR is higher-is-better (↑).
• Chinese benchmark excludes models without Chinese response capabilities (e.g., Mini-Omni variants).

Evidence (verbatim from paper)

In SpeechInstructBench, evaluation metrics encompass prompt-level (P) and instruction-level (I) accuracy for both closed-ended and open-ended tasks. For closed-ended questions, P and I metrics are computed by averaging loose and strict accuracies, while adjustment task is assessed using Instruction Adherence Rate (IAR) and Error Correction Rate (ECR).

Citation

@misc{wang2025inserter,
  title={InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.02769}
}

• arXiv: 2503.02769