P2v Audio Deepfake Eval

p2v-audio-deepfake-eval

Perturbed Public Voices (P$^{2}$V): A Dataset for Robust Audio Deepfake Detection — Gao et al. (2025) (arXiv:2508.10949, 2025)

What this evaluates

This benchmark evaluates the robustness and cross-dataset generalization of audio deepfake detection models under realistic acoustic perturbations and across diverse state-of-the-art voice cloning and TTS methods. It probes whether detectors learn genuine synthetic speech artifacts or overfit to dataset-specific biases like unusual dialogue or background noise.

Datasets

• P2V (Perturbed Public Voices) — total 257440; splits: test (-1)
• In-The-Wild (ITW) — total 31779; splits: train (-1), val (-1), test (-1)

Metrics

• DDS (Deepfake Detection Score) (primary) — range: percent
  • DDS = (F1_fake + AUC + (1 - EER)) / 3. All reported values are scaled by a factor of 100 for ease of interpretation.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the model's ability to distinguish between real and deepfake audio across all classification thresholds.
• EER — range: [0, 1]
  • Equal Error Rate, the operating point where the false acceptance rate equals the false rejection rate.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Overall precision and recall are computed using weighted averages based on class instance counts to handle imbalance.

Input / output format

Input: Audio files (raw waveform or extracted features such as LFCC, MFCC, or Whisper embeddings) representing speech samples labeled as either authentic or deepfake.

Output: Binary classification prediction (probability or logit indicating the likelihood of the audio being a deepfake), typically thresholded at 0.5 for metric computation.

Scoring recipe

def compute_metrics(y_true, y_pred_prob):
    y_pred = (y_pred_prob >= 0.5).astype(int)
    f1_fake = f1_score(y_true, y_pred, pos_label=1)
    auc = roc_auc_score(y_true, y_pred_prob)
    eer = compute_eer(y_true, y_pred_prob)
    dds = ((f1_fake + auc + (1 - eer)) / 3) * 100
    return {'F1_fake': f1_fake, 'AUC': auc, 'EER': eer, 'DDS': dds}

Common pitfalls

• Reported metrics (except EER) are scaled by 100 (e.g., DDS=60.82), which can cause confusion if interpreted as raw probabilities.
• Overall precision and recall use weighted averages to account for class imbalance, not macro averages.
• Models frequently overfit to dataset-specific characteristics (e.g., ITW's absurd dialogue or natural noise) rather than learning generalizable deepfake artifacts, leading to poor cross-dataset transfer.

Evidence (verbatim from paper)

The evaluation metrics include precision, recall, and F1 scores for each class individually. We also compute overall precision and recall using weighted averages, where the number of instances per class determines the weights. In addition, we report AUC and Equal Error Rate (EER), a widely used metric in deepfake detection that captures the balance point between false acceptance and false rejection rates. We compute an aggregated Deepfake Detection Score (DDS) to provide a concise summary of the model’s overall performance: DDS=(F1_{Fake}+AUC+(1-EER))/3. For ease of interpretation, all reported metrics (except for EER) are scaled by a factor of 100.

Citation

@misc{gao2025p2v,
  title={Perturbed Public Voices (P$^{2}$V): A Dataset for Robust Audio Deepfake Detection},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2508.10949}
}

• arXiv: 2508.10949