Cloned Voice Detection Eval

cloned-voice-detection-eval

Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features — Barrington et al. (2023) (arXiv:2307.07683, 2023)

What this evaluates

Evaluates the ability of audio classifiers to distinguish between real human speech and AI-generated cloned voices across single and multi-speaker scenarios. It also probes robustness against adversarial audio laundering, including additive noise and AAC transcoding, to assess how well different feature representations (learned, spectral, perceptual) generalize and resist degradation.

Datasets

• ElevenLabs (EL) — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection
• Uberduck (UD) — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection
• WaveFake (WF) — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection
• TIMIT-ElevenLabs — total ?; splits: test (-1); repo https://github.com/audio-df-ucb/ClonedVoiceDetection

Metrics

• EER (%) (primary) — range: percent
  • Equal Error Rate is the point on the receiver operating characteristic (ROC) curve where the false acceptance rate (FAR, incorrectly classifying a synthetic voice as real) equals the false rejection rate (FRR, incorrectly classifying a real voice as synthetic).
• Synthetic Accuracy (%) — range: percent
  • Percentage of synthetic audio samples correctly classified as synthetic.
• Real Accuracy (%) — range: percent
  • Percentage of real audio samples correctly classified as real.

Input / output format

Input: Raw audio samples labeled as either real human speech or AI-generated synthetic speech (optionally subjected to adversarial laundering like additive Gaussian noise or AAC transcoding).

Output: Binary classification label (real vs. synthetic) or multi-class label (real vs. specific synthesis engine). Accuracy is computed separately for synthetic and real classes, with EER reported for single-class classifiers.

Scoring recipe

def compute_metrics(predictions, synth_mask, real_mask):
    synth_acc = sum(predictions[synth_mask]) / synth_mask.sum()
    real_acc = sum(1 - predictions[real_mask]) / real_mask.sum()
    
    # EER calculation
    thresholds = np.linspace(0, 1, 1000)
    eer = None
    for t in thresholds:
        far = sum(predictions[synth_mask] > t) / synth_mask.sum()
        frr = sum(predictions[real_mask] <= t) / real_mask.sum()
        if abs(far - frr) < 1e-3:
            eer = (far + frr) / 2
            break
    return synth_acc, real_acc, eer

Common pitfalls

• Laundering (additive noise + AAC transcoding) significantly degrades spectral and perceptual features more than learned features, skewing comparisons if laundering is not controlled.
• Classifiers trained on a single dataset or synthesis engine do not generalize well to others; multi-dataset training is required for robust detection.
• EER is only reported for single-class classifiers; multi-class setups report separate per-class accuracies instead of EER.

Evidence (verbatim from paper)

The far-right columns report the equal error rate (the EER is the point on the receiver operating curve (ROC) where the false acceptance rate (incorrectly classifying a synthetic voice as real) and false rejection rate (incorrectly classifying a real voice as synthetic) are equal).

Citation

@misc{barrington2023clonedvoice,
  title={Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features},
  author={Barrington et al. (2023)},
  year={2023},
  note={arXiv:2307.07683}
}

• arXiv: 2307.07683