Fish Audio S2 Eval

fish-audio-s2-eval

Fish Audio S2 Technical Report — Liao et al. (2026) (arXiv:2603.08823, 2026)

What this evaluates

Evaluates speech synthesis models on intelligibility, speaker similarity, and long-form generation across multiple languages. It also assesses subjective qualities like naturalness, instruction-following, and human-level indistinguishability using automated LLM-as-a-Judge and Audio Turing Test frameworks.

Datasets

• Seed-TTS-Eval — total ?; splits: test-zh (-1), test-en (-1), zh-hard (-1)
• CV3-Eval — total ?; splits: multilingual-9lang (-1)
• Minimax Multilingual Testset — total ?; splits: 24-languages (-1)
• Long-TTS-Eval — total ?; splits: English (-1), Chinese (-1)
• Audio Turing Test — total 499; splits: ATT-Corpus (499)
• Emergent TTS Eval — total ?; splits: 5-dimensions (-1)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate: ratio of edit operations (insertions, deletions, substitutions) between the ASR-transcribed output and the reference text.
• CER (%) — range: percent
  • Character Error Rate: same as WER but computed at the character level, primarily used for Chinese evaluation.
• SIM (Speaker Similarity) — range: [0, 1]
  • Cosine similarity between speaker embeddings extracted from generated audio (3s chunks, 1.5s hop) and reference audio using WavLM-large. Reported as mean and standard deviation across chunks.
• Win-Rate (%) — range: percent
  • Percentage of pairwise comparisons where the model's output is preferred over a baseline model by the LLM-as-a-Judge or evaluation framework.
• ATT Posterior Mean — range: [0, 1]
  • Posterior probability of the 'Human' class from the Audio Turing Test evaluation, estimated via Bayesian inference using the Auto-ATT model.

Input / output format

Input: Reference audio clip and text prompt (optionally with instruction/style tags).

Output: Synthesized audio waveform.

Scoring recipe

def compute_wer_cer(generated_audio, ref_text, lang):
    asr = 'Whisper-large-v3' if lang == 'en' else 'Paraformer-zh'
    transcribed = asr.transcribe(generated_audio, chunk_size=28)
    return edit_distance(transcribed, ref_text) / len(ref_text)

def compute_sim(gen_audio, ref_audio):
    chunks = chunk_audio(gen_audio, duration=3.0, hop=1.5)
    ref_emb = wavlm_large.encode(ref_audio)
    sims = [cosine_similarity(wavlm_large.encode(c), ref_emb) for c in chunks]
    return {'mean': mean(sims), 'std': std(sims)}

def compute_att_posterior(audio):
    return auto_att_model.predict_proba(audio)['Human']

Common pitfalls

• WER/CER values are highly dependent on the specific ASR model (Whisper vs Paraformer) and chunking strategy (28s non-overlapped) used for transcription.
• SIM scores require strict adherence to the 3s chunking with 1.5s hop and WavLM-large embeddings; deviating from this changes the scale.
• ATT results are reported as posterior means from a Bayesian model, not simple classification accuracy, making direct comparison with standard accuracy metrics invalid.

Evidence (verbatim from paper)

We assess voice-cloning intelligibility on Seed-TTS-Eval using WER over the test-zh, test-en, and ZH-hard splits. WER is computed by transcribing synthesized audio with Whisper-large-v3 (Radford et al., 2023) for English and Paraformer-zh Gao et al. (2023) for Chinese, following the benchmark protocol; results are reported in Table 1.

Citation

@misc{liao2026fishaudios2,
  title={Fish Audio S2 Technical Report},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2603.08823}
}

• arXiv: 2603.08823