# Foice Detection Eval

> Evaluates the ability of state-of-the-art audio deepfake detectors to distinguish real speech from face-to-voice (FOICE) synthesized speech, and assesses how fine-tuning on FOICE data affects robustness against unseen synthesis pipelines like SpeechT5. Use when the user wants to benchmark on FOICE, SpeechT5, or asks about evaluating this task. Reports EER.

- Skill: `qhjqhj00/foice-detection-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/foice-detection-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/foice-detection-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/foice-detection-eval

---


# foice-detection-eval

> Can Current Detectors Catch Face-to-Voice Deepfake Attacks? — Nguyen et al. (2025) (arXiv:2510.21004, 2025)

## What this evaluates

Evaluates the ability of state-of-the-art audio deepfake detectors to distinguish real speech from face-to-voice (FOICE) synthesized speech, and assesses how fine-tuning on FOICE data affects robustness against unseen synthesis pipelines like SpeechT5.

## Datasets

- **FOICE** — total ?; splits: test (-1)
- **SpeechT5** — total ?; splits: test (-1)

## Metrics

- `EER` **(primary)** — range: [0, 1]
  - Equal Error Rate: the operating point where False Acceptance Rate (FAR) equals False Rejection Rate (FRR). Lower values indicate better discrimination.
- `F1-Score` — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Higher values indicate better balance between sensitivity and specificity.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified samples (TP + TN) out of total samples.
- `Precision` — range: [0, 1]
  - Proportion of predicted positives that are actual positives: TP / (TP + FP).
- `Recall` — range: [0, 1]
  - Proportion of actual positives correctly identified: TP / (TP + FN).

## Input / output format

**Input**: Raw audio samples (real speech or synthesized speech from FOICE/SpeechT5 pipelines), optionally corrupted with noise or passed through a denoiser.

**Output**: Binary classification decision (real vs. fake) or confidence scores used to compute operating points for EER and standard classification metrics.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    tp = sum(p == 1 and l == 1 for p, l in zip(predictions, labels))
    fp = sum(p == 1 and l == 0 for p, l in zip(predictions, labels))
    fn = sum(p == 0 and l == 1 for p, l in zip(predictions, labels))
    tn = sum(p == 0 and l == 0 for p, l in zip(predictions, labels))
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    eer = calculate_eer(predictions, labels)
    return {'EER': eer, 'Accuracy': accuracy, 'Precision': precision, 'Recall': recall, 'F1-Score': f1}
```

## Common pitfalls

- Assuming detectors trained on standard vocoder-based datasets (ASVspoof, WaveFake) will generalize to FOICE, as FOICE lacks explicit vocoder residue cues.
- Equating high in-distribution fine-tuning performance with robust generalization; fine-tuning on FOICE causes severe accuracy drops on unseen pipelines like SpeechT5.
- Ignoring audio conditions (clean, noisy, denoised) when comparing detector robustness, as performance varies significantly across these settings.

## Evidence (verbatim from paper)

> Baseline detectors (denoted "Base" in the Model column) show limited discriminative ability on FOICE-generated speech, with high EERs and inconsistent accuracy across clean, noisy, and denoised audio, as shown in Table[I], Figures[2] and [5] in Appendix[A].

## Citation

```bibtex
@misc{nguyen2025face2voicedetection,
  title={Can Current Detectors Catch Face-to-Voice Deepfake Attacks?},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2510.21004}
}
```

- arXiv: 2510.21004