Echofake Eval

echofake-eval

EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detection — Zhang et al. (2025) (arXiv:2510.19414, 2025)

What this evaluates

Probes the robustness of speech deepfake detection models against physical replay attacks and cross-dataset generalization. It evaluates how well anti-spoofing systems distinguish between genuine speech, zero-shot TTS-generated deepfakes, and their physically replayed counterparts under realistic acoustic conditions.

Datasets

• EchoFake — total ?; splits: train (-1), open (-1); repo https://github.com/EchoFake/EchoFake

Metrics

• EER (primary) — range: percent
  • Equal Error Rate: the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). Reported as a percentage.
• F1-score — range: percent
  • Harmonic mean of precision and recall. Evaluated for four-class classification (B/RB/F/RF) and averaged across classes.

Input / output format

Input: Raw audio recordings (speech samples) with ground-truth labels.

Output: Binary label (spoof/bonafide) or four-class label (B/RB/F/RF).

Scoring recipe

def compute_eer(pred_scores, true_labels):
    # pred_scores: probability of spoof, true_labels: 1=spoof, 0=bonafide
    thresholds = np.linspace(0, 1, 1000)
    best_diff = float('inf')
    eer = 0.0
    for t in thresholds:
        far = sum(s >= t and l == 0 for s, l in zip(pred_scores, true_labels)) / max(sum(l == 0 for l in true_labels), 1)
        frr = sum(s < t and l == 1 for s, l in zip(pred_scores, true_labels)) / max(sum(l == 1 for l in true_labels), 1)
        if abs(far - frr) < best_diff:
            best_diff = abs(far - frr)
            eer = (far + frr) / 2
    return eer

Common pitfalls

• Treating Replayed Bona Fide (RB) samples as genuine in binary evaluation; the protocol explicitly treats RB as spoofed to test replay robustness.
• Ignoring the distinction between closed-set and open-set conditions; performance drops sharply in open-set scenarios where replayed samples are the primary error source.
• Assuming spectral artifacts are the sole detection cue; high-fidelity replay attacks mask these artifacts, requiring models to learn channel distortion features.

Evidence (verbatim from paper)

We evaluate system performance using two standard metrics: F1-score for four-class classification (B/RB/F/RF) and equal error rate (EER) for binary spoofing detection (spoof/bonafide). The F1-score provides balanced accuracy measurement, while EER optimizes the trade-off between false acceptance and rejection rates.

Citation

@misc{zhang2025echofake,
  title={EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detection},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.19414}
}

• arXiv: 2510.19414