Foice Detection Eval

foice-detection-eval

Can Current Detectors Catch Face-to-Voice Deepfake Attacks? — Nguyen et al. (2025) (arXiv:2510.21004, 2025)

What this evaluates

Evaluates the ability of state-of-the-art audio deepfake detectors to distinguish real speech from face-to-voice (FOICE) synthesized speech, and assesses how fine-tuning on FOICE data affects robustness against unseen synthesis pipelines like SpeechT5.

Datasets

• FOICE — total ?; splits: test (-1)
• SpeechT5 — total ?; splits: test (-1)

Metrics

• EER (primary) — range: [0, 1]
  • Equal Error Rate: the operating point where False Acceptance Rate (FAR) equals False Rejection Rate (FRR). Lower values indicate better discrimination.
• F1-Score — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Higher values indicate better balance between sensitivity and specificity.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified samples (TP + TN) out of total samples.
• Precision — range: [0, 1]
  • Proportion of predicted positives that are actual positives: TP / (TP + FP).
• Recall — range: [0, 1]
  • Proportion of actual positives correctly identified: TP / (TP + FN).

Input / output format

Input: Raw audio samples (real speech or synthesized speech from FOICE/SpeechT5 pipelines), optionally corrupted with noise or passed through a denoiser.

Output: Binary classification decision (real vs. fake) or confidence scores used to compute operating points for EER and standard classification metrics.

Scoring recipe

def compute_metrics(predictions, labels):
    tp = sum(p == 1 and l == 1 for p, l in zip(predictions, labels))
    fp = sum(p == 1 and l == 0 for p, l in zip(predictions, labels))
    fn = sum(p == 0 and l == 1 for p, l in zip(predictions, labels))
    tn = sum(p == 0 and l == 0 for p, l in zip(predictions, labels))
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    eer = calculate_eer(predictions, labels)
    return {'EER': eer, 'Accuracy': accuracy, 'Precision': precision, 'Recall': recall, 'F1-Score': f1}

Common pitfalls

• Assuming detectors trained on standard vocoder-based datasets (ASVspoof, WaveFake) will generalize to FOICE, as FOICE lacks explicit vocoder residue cues.
• Equating high in-distribution fine-tuning performance with robust generalization; fine-tuning on FOICE causes severe accuracy drops on unseen pipelines like SpeechT5.
• Ignoring audio conditions (clean, noisy, denoised) when comparing detector robustness, as performance varies significantly across these settings.

Evidence (verbatim from paper)

Baseline detectors (denoted "Base" in the Model column) show limited discriminative ability on FOICE-generated speech, with high EERs and inconsistent accuracy across clean, noisy, and denoised audio, as shown in Table[I], Figures[2] and [5] in Appendix[A].

Citation

@misc{nguyen2025face2voicedetection,
  title={Can Current Detectors Catch Face-to-Voice Deepfake Attacks?},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2510.21004}
}

• arXiv: 2510.21004