Versebench Eval

versebench-eval

MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning — Qiang et al. (2026) (arXiv:2601.01568, 2026)

What this evaluates

Evaluates a unified multimodal model's ability to generate synchronized audio and video from text, phonemes, and reference media. It probes zero-shot voice cloning fidelity, lip-sync accuracy, acoustic quality, and cross-modal temporal alignment.

Datasets

• VerseBench — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed by transcribing generated audio with Whisper-large-v3 and comparing against ground-truth text.
• SIM-o — range: [0, 1]
  • Speaker similarity score calculated using the WavLM-large speaker verification model following the SeedTTS protocol.
• SNC — range: other
  • SyncNet Confidence score measuring lip-synchronization quality between generated audio and video frames.
• CLAP — range: [0, 1]
  • Semantic alignment score between audio and text using the CLAP model.
• FD — range: other
  • Fréchet Distance on mel-spectrogram features extracted via PANNs, measuring distributional distance.
• AS — range: [0, 1]
  • Aesthetic Score derived from MANIQA, aesthetic-predictor-v2-5, and Musiq.
• MS — range: [0, 1]
  • Motion Score based on RAFT optical flow to quantify dynamic realism.
• ID — range: [0, 1]
  • Identity Consistency measured as mean DINOV3 feature similarity between reference and generated frames.
• AV-A — range: other
  • Audio-Video Alignment distance computed via Synchformer.
• IB — range: [0, 1]
  • ImageBind score measuring global cross-modal alignment.

Input / output format

Input: Text instructions, phoneme sequences, and optionally reference audio (for timbre cloning) and reference images (for animation).

Output: Synchronized video frames and corresponding audio waveform.

Scoring recipe

def compute_metrics(predictions, gold):
    wer = whisper_transcribe(predictions.audio).wer(gold.text)
    sim_o = wavlm_similarity(predictions.audio, gold.ref_audio)
    snc = syncnet_confidence(predictions.audio, predictions.video)
    clap = clap_score(predictions.audio, gold.text)
    fd = frechet_distance(melspectrogram(predictions.audio), melspectrogram(gold.audio))
    return {'WER': wer, 'SIM-o': sim_o, 'SNC': snc, 'CLAP': clap, 'FD': fd}

Common pitfalls

• Comparing joint audio-video generation models against audio-only TTS baselines without acknowledging the increased generative complexity of the multimodal task.
• Using different speaker verification backends (WavLM-large for SIM-o vs. Resemblyzer for ablation studies), which prevents direct numerical comparison of identity preservation scores across tables.
• Relying solely on automated metrics for lip-sync and identity, as the paper also supplements these with human preference tests and pass-rate thresholds.

Evidence (verbatim from paper)

Speech intelligibility is quantified by the Word Error Rate (WER), derived from Whisper-large-v3 transcriptions. To assess speaker identity preservation, we employ specific metrics depending on the evaluation context. For the main comparative benchmark against state-of-the-art TTS models (Table[2]), we follow the SeedTTS evaluation protocol and report SIM-o, calculated using the WavLM-large speaker verification model.

Citation

@misc{qiang2026mmsonate,
  title={MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning},
  author={Qiang et al. (2026)},
  year={2026},
  note={arXiv:2601.01568}
}

• arXiv: 2601.01568