Cloneshield Eval

cloneshield-eval

CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning — Li et al. (2025) (arXiv:2505.19119, 2025)

What this evaluates

Evaluates a universal adversarial perturbation framework designed to defend against zero-shot voice cloning TTS models. It measures how well the perturbation degrades cloned audio quality and speaker similarity while preserving the perceptual fidelity of the original protected speech.

Datasets

• VCTK — total 450; splits: test (450)
• LibriSpeech ASR — total 450; splits: test (450)
• LibriTTS-R — total 450; splits: test (450)
• LJSpeech — total 450; splits: test (450)
• Common Voice — total 450; splits: test (450)

Metrics

• SRS — range: [0, 1]
  • Speaker Recognition Similarity computed using the Pyannote.audio model to measure similarity between original and adversarial/cloned audio.
• PESQ — range: other
  • Perceptual Evaluation of Speech Quality, a standard objective metric for assessing the naturalness and quality of speech signals.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility, a measure of perceptual-level sample-level clarity and speech intelligibility.
• CER — range: [0, 1]
  • Character Error Rate computed using SpeechRecognition 3.11.0 to measure transcription accuracy of inputs and synthesized outputs.
• DSR (primary) — range: [0, 1]
  • Defense Success Rate, defined as successful if the similarity score of the synthesized result falls below 0.50.

Input / output format

Input: Raw audio utterances (450 randomly selected per dataset) paired with distinct text prompts assigned per dataset to drive zero-shot TTS synthesis.

Output: Synthesized audio from the target TTS model, plus the perturbed input audio. Metrics are computed on both the protected inputs and the cloned outputs.

Scoring recipe

def compute_evaluation(original_audio, perturbed_audio, cloned_audio, target_text):
    # Input quality & preservation
    pesq = compute_pesq(original_audio, perturbed_audio)
    stoi = compute_stoi(original_audio, perturbed_audio)
    srs_in = compute_srs(original_audio, perturbed_audio)
    cer1 = compute_cer(original_audio, target_text)
    cer2 = compute_cer(perturbed_audio, target_text)
    
    # Output defense effectiveness
    cer_out = compute_cer(cloned_audio, target_text)
    srs_out = compute_srs(cloned_audio, original_audio)
    dsr = 1.0 if srs_out < 0.50 else 0.0
    
    return {
        'PESQ': pesq, 'STOI': stoi, 'SRS_in': srs_in,
        'CER1': cer1, 'CER2': cer2, 'CER_out': cer_out,
        'SRS_out': srs_out, 'DSR': dsr
    }

Common pitfalls

• DSR threshold is defined as <0.50 for success, which is counterintuitive compared to standard accuracy metrics where higher is better.
• Evaluation uses batch-optimized perturbations (up to 5 utterances jointly optimized), making it significantly harder than single-utterance defenses and affecting generalization metrics.
• Metric directions vary across the suite: lower is better for CER, MCD, and LSD, while higher is better for PESQ, STOI, SNR, and SRS.

Evidence (verbatim from paper)

We use Pyannote.audio model (Bredin et al., [2020]) to conduct Speaker Recognition Similarity (SRS) between original data and adversarial data. Signal-to-Distortion Ratio (SDR) (Yamamoto et al., [2017]) is applied for waveform distortion metric. We use Log Spectral Distance (LSD) (Swamy et al., [2020]) and Mel Cepstral Distortion (MCD) (Brandt et al., [2017]) as the metrics for estimating spectral similarity, to evaluate the destructiveness of adversarial data on timbre. The Perceptual Evaluation of Speech Quality (PESQ) (Martin-Donas et al., [2018]) is employed to indicate naturalness of voice. Short-Time Objective Intelligibility (STOI) (Andersen et al., [2017]) and Signal-to-Noise Ratio (SNR) (Peng et al., [2020]) are measures of perceptual-Level sample-level clarity intelligibility. We additionally apply SpeechRecognition 3.11.0 to count Character Error Rate (CER) of inputs and synthesized results. In addition, we also introduce the defense success rate to measure CloneShield’s generalizability on different speakers. We define a defense as successful if the DSR of the synthesized result falls below 0.50.

Citation

@misc{li2025cloneshield,
  title={CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.19119}
}

• arXiv: 2505.19119