Rvcbench Eval

rvcbench-eval

RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models — Liao et al. (2026) (arXiv:2602.00443, 2026)

What this evaluates

Evaluates the robustness of modern voice cloning models under realistic deployment conditions, including input variations (accents, text shifts, long context), cross-lingual synthesis, post-processing degradation, and adversarial perturbations. It probes the trade-offs between generation quality, content fidelity, speaker similarity, and deepfake detectability across diverse acoustic and linguistic stressors.

Datasets

• LibriTTS — total ?; splits: test (-1)
• VCTK — total ?; splits: test (-1)
• LibriSpeech — total ?; splits: test (-1)
• RVCBench — total 14370; splits: multilingual (-1), audio_shift (-1), text_shift (-1), expression (-1), compression (-1), detectability (-1), anti_protect (-1); repo https://github.com/Nanboy-Ronan/RVCBench

Metrics

• SIM (primary) — range: [0, 1]
  • Speaker similarity score computed via cosine similarity of pre-trained speaker embeddings between reference and synthesized audio.
• WER (primary) — range: percent
  • Word Error Rate measuring transcription accuracy of the synthesized audio against the target text using an ASR system.
• MCD (primary) — range: other
  • Mel-cepstral Distortion quantifying the spectral difference between reference and synthesized speech frames.
• MOS — range: [1, 5]
  • Mean Opinion Score assessing perceptual audio quality and naturalness via human or LLM-as-Judge ratings.
• EER — range: percent
  • Equal Error Rate from a deepfake detector (SQ-LLM) where false acceptance and false rejection rates intersect.
• minDCF — range: percent
  • Minimum Detection Cost Function evaluating the trade-off between false acceptance and false rejection costs for deepfake detection.
• ACC — range: percent
  • Classification accuracy of the deepfake detector in distinguishing ground-truth, benign cloned, and scam cloned speech.

Input / output format

Input: Reference audio clip (speaker voice), target text prompt, and optional perturbation parameters (e.g., noise type, compression codec, adversarial method, language).

Output: Synthesized audio waveform matching the target text in the reference speaker's voice.

Scoring recipe

def compute_metrics(ref_audio, gen_audio, target_text, det_model):
    sim = cosine_similarity(speaker_embed(ref_audio), speaker_embed(gen_audio))
    wer = asr_transcribe(gen_audio).word_error_rate(target_text)
    mcd = mel_cepstral_distortion(ref_audio, gen_audio)
    mos = llm_judge(gen_audio).score()
    det_logits = det_model(gen_audio)
    eer, mindcf = compute_det_curve(det_logits, is_fake_label)
    acc = accuracy(det_logits, is_fake_label)
    return {'SIM': sim, 'WER': wer, 'MCD': mcd, 'MOS': mos, 'EER': eer, 'minDCF': mindcf, 'ACC': acc}

Common pitfalls

• Assuming high generation quality (MOS/SIM) implies strong evasion from deepfake detectors; the paper shows competitive models like FishSpeech remain highly detectable.
• Believing longer reference audio always improves robustness; performance gains plateau after 8–12 seconds and can introduce non-monotonic instability.
• Overlooking that cross-lingual tasks primarily bottleneck on content accuracy (WER) rather than speaker similarity, despite some models maintaining high SIM.

Evidence (verbatim from paper)

We assess deepfake detectability against impersonation using EER, minDCF, and ACC, and find detectability varies across VCmodels in Tab.[3]. Higher ACC and lower EER/minDCF denote more deepfake perceptible.

Citation

@misc{liao2026rvcbench,
  title={RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2602.00443}
}

• arXiv: 2602.00443