Unisonte Audio Eval

unisonte-audio-eval

UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions — Qiang et al. (2026) (arXiv:2604.22209, 2026)

What this evaluates

Evaluates a unified text-to-audio model's ability to generate speech, music, and sound effects from natural language instructions without reference audio. It probes instruction-following fidelity, acoustic quality, structural coherence, and the positive transfer effects of multi-modal joint training.

Datasets

• UniSonate Unified Corpus — total ?; splits: (unstated)
• Seed-TTS test set — total ?; splits: test (-1)
• SongEval benchmark — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate measures the percentage of incorrectly recognized words relative to the ground truth transcript, calculated as the minimum number of insertions, deletions, and substitutions divided by the total number of words in the reference.
• SongEval (primary) — range: other
  • A composite benchmark score evaluating generated music across five dimensions: Coherence, Musicality, Memorability, Clarity, and Naturalness, typically rated on a Likert scale by human listeners or automated evaluators.
• FAD — range: other
  • Fréchet Audio Distance computes the Fréchet distance between the multivariate Gaussian distributions of features extracted from a pre-trained audio model for generated and real audio samples.
• CLAP score — range: [0, 1]
  • The cosine similarity between the audio embedding and the corresponding text instruction embedding in the CLAP (CLIP-Like Audio Pretraining) latent space.
• NMOS — range: other
  • Natural Mean Opinion Score derived from subjective listening tests where human raters evaluate audio quality on a standard 1-5 scale without reference audio.

Input / output format

Input: Natural language instruction text describing acoustic attributes (e.g., gender, emotion, genre, or event). For speech and music, phoneme sequences are additionally provided via a frozen Zipformer encoder.

Output: 44.1kHz audio waveform with durations ranging from 2 to 20 seconds.

Scoring recipe

def evaluate(predictions, gold):
    wer = edit_distance(transcribe(predictions), gold) / len(gold.split())
    fad = frechet_distance(audio_features(predictions), audio_features(gold))
    clap = cosine_similarity(audio_embed(predictions), text_embed(gold))
    mos = average(subjective_ratings(predictions))
    return {'WER': wer, 'FAD': fad, 'CLAP': clap, 'MOS': mos}

Common pitfalls

• Comparing reference-free generation directly against reference-based models on raw MOS scores without accounting for the inherent advantage of reference audio guidance.
• Assuming multi-modal joint training degrades single-modality performance; ablation studies show joint training actually improves WER and structural coherence via positive transfer.
• Evaluating long-form structural coherence without acknowledging the strict 2–20 second clip duration constraint used during both training and testing.

Evidence (verbatim from paper)

Table [3] reports the Word Error Rate (WER) on the Seed-TTS test set. UniSonate achieves the lowest WER (1.47% on English and 1.25% on Chinese), surpassing both the dedicated TTS baselines (e.g., F5-TTS, CosyVoice2) and the previous unified model InstructAudio. This suggests that the inclusion of diverse audio data (music and sound effects) during the curriculum learning phase does not dilute speech intelligibility; rather, it appears to enhance the model’s acoustic robustness.

Citation

@misc{qiang2026unisonte,
  title={UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions},
  author={Qiang et al. (2026)},
  year={2026},
  note={arXiv:2604.22209}
}

• arXiv: 2604.22209