Instructaudio Eval

instructaudio-eval

InstructAudio: Unified speech and music generation with natural language instruction — Qiang et al. (2025) (arXiv:2511.18487, 2025)

What this evaluates

This evaluation probes a model's ability to generate speech and music conditioned on natural language instructions describing acoustic and musical attributes. It measures text-to-audio fidelity, attribute control accuracy, and perceptual quality across short-form generation tasks.

Datasets

• Seed-TTS benchmark — total ?; splits: test (-1)
• InstructAudio internal test set — total 500; splits: test (500)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: percentage of incorrectly recognized words relative to the reference transcript, computed via edit distance alignment.
• Classification Control Accuracy — range: percent
  • Percentage of generated samples where a classifier correctly identifies the target attribute (e.g., gender, age, emotion) matching the instruction.
• SongEval — range: other
  • Benchmark scoring coherence, musicality, memorability, clarity, and naturalness on a 1-5 scale based on human listening tests.
• QMOS/NMOS/MMOS — range: other
  • Mean Opinion Scores from trained evaluators rating audio quality, naturalness, or musicality respectively.

Input / output format

Input: Natural language instruction specifying target attributes (e.g., gender, age, emotion, style, accent for speech; genre, instrument, rhythm, atmosphere for music), optionally paired with ground truth text/lyrics.

Output: 44.1kHz audio waveform clip (2–20 seconds).

Scoring recipe

def compute_wer(predictions, gold):
    alignment = edit_distance_align(predictions, gold)
    errors = sum(alignment.substitutions + alignment.deletions + alignment.insertions)
    total = len(gold.split())
    return (errors / total) * 100 if total > 0 else 0.0

def compute_control_accuracy(predictions, gold_attributes):
    correct = 0
    for pred_audio, target_attr in zip(predictions, gold_attributes):
        predicted_attr = classifier.predict(pred_audio)
        if predicted_attr == target_attr:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Using reference audio for baseline models (e.g., CosyVoice2) while InstructAudio uses text-only conditioning creates an unfair timbre control comparison.
• Truncating long music generations to 5–20s clips biases evaluation against models optimized for full-length tracks.
• Short clip duration limits assessment of temporal coherence and long-form structural quality.

Evidence (verbatim from paper)

We employ comprehensive objective and subjective metrics to ensure thorough evaluation. Objective metrics include Word Error Rate (WER) using Seed-TTS, Speaker Similarity, Emotion Similarity, Log-Spectral Distance (LSD), Mel-Cepstral Distortion (MCD), Mean Squared Error of Pitch (MSEP), Voiced/Unvoiced Mismatch Rate (MR), SongEval music evaluation benchmark, and classification control accuracy through perceptual consistency assessment.

Citation

@misc{qiang2025instructaudio,
  title={InstructAudio: Unified speech and music generation with natural language instruction},
  author={Qiang et al. (2025)},
  year={2025},
  note={arXiv:2511.18487}
}

• arXiv: 2511.18487