Arfake Eval

arfake-eval

ArFake: A Multi-Dialect Benchmark and Baselines for Arabic Spoof-Speech Detection — Maged et al. (2025) (arXiv:2509.22808, 2025)

What this evaluates

Evaluates the ability of spoof-speech detection models to distinguish between real (bonafide) Arabic speech and synthetic speech generated by various Text-to-Speech (TTS) models across multiple Arabic dialects. It probes robustness against different voice-cloning systems and measures detection accuracy alongside perceptual realism and ASR fidelity.

Datasets

• ArFake — total ?; splits: train (-1), test (23000)

Metrics

• Equal Error Rate (EER) (primary) — range: percent
  • Computed by sweeping a decision threshold to find the operating point where False Acceptance Rate (FAR) equals False Rejection Rate (FRR). Lower values indicate better detection performance.
• Accuracy — range: percent
  • The percentage of correctly classified samples (bonafide vs. spoofed) out of the total test set.

Input / output format

Input: Audio samples (bonafide or spoofed) with corresponding class labels. For ASR evaluation, audio paired with ground-truth transcriptions.

Output: Binary classification prediction (bonafide or spoofed) for detection models. Transcription text for ASR models. Ratings from 1 to 5 for MOS evaluation.

Scoring recipe

def compute_eer_and_acc(predictions, labels):
    tp = sum(p == l == 1 for p, l in zip(predictions, labels))
    fp = sum(p == 1 and l == 0 for p, l in zip(predictions, labels))
    fn = sum(p == 0 and l == 1 for p, l in zip(predictions, labels))
    tn = sum(p == l == 0 for p, l in zip(predictions, labels))
    far = fp / (fp + tn) if (fp + tn) > 0 else 0
    frr = fn / (fn + tp) if (fn + tp) > 0 else 0
    # EER is the threshold where FAR = FRR (typically found via continuous score sweep)
    acc = (tp + tn) / len(labels)
    return eer, acc

Common pitfalls

• EER is undefined for single-class test splits (e.g., VITS-only data), so Accuracy must be used as the fallback metric.
• Models trained on a single TTS generator show artificially low EER compared to the combined multi-generator training protocol, reducing claims of generalizability.
• MOS evaluation uses only 8 samples per TTS model (1 per dialect) rated by 12 speakers, which may not capture full acoustic or dialectal variability.

Evidence (verbatim from paper)

We evaluated model performance using Accuracy and Equal Error Rate (EER) evaluation metrics. The lower EER values indicate more reliable detectors and thus receive greater focus in our analysis.

Citation

@misc{maged2025arfake,
  title={ArFake: A Multi-Dialect Benchmark and Baselines for Arabic Spoof-Speech Detection},
  author={Maged et al. (2025)},
  year={2025},
  note={arXiv:2509.22808}
}

• arXiv: 2509.22808