Hula Antispoofing Eval

hula-antispoofing-eval

HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech — Mahapatra et al. (2025) (arXiv:2509.21676, 2025)

What this evaluates

This benchmark evaluates anti-spoofing systems for synthetic speech detection, with a specific focus on prosody-awareness, emotional/expressive spoofing, and cross-lingual robustness. It tests whether models can distinguish real speech from TTS, VC, and adversarial attacks across diverse channel conditions and languages.

Datasets

• ASVspoof 2019 (LA) — total 71237; splits: test (71237)
• ASVspoof 2021 (LA) — total 148176; splits: test (148176)
• ASVspoof 2024 (Track 1) — total 680774; splits: test (680774)
• EmoFake — total 17500; splits: test (17500)
• Mixed Emotions — total 491; splits: test (491)
• ADD 2022 (Track 1) — total 109199; splits: test (109199)
• HABLA — total 75816; splits: test (75816)

Metrics

• EER% (primary) — range: percent
  • Equal Error Rate (EER) is the operating point where the False Acceptance Rate (FAR) and False Rejection Rate (FRR) are equal. It is reported as a percentage.

Input / output format

Input: Audio clips trimmed or zero-padded to approximately 4 seconds. Processed through an XLS-R 300M backbone with 25 ms frame length and 20 ms frame period.

Output: Binary spoof detection score (probability of being synthetic), thresholded to compute EER%.

Scoring recipe

def compute_eer(scores, labels):
    # scores: float array (higher = more likely spoof)
    # labels: 1 for spoof, 0 for bona fide
    far, frr = compute_roc_curve(scores, labels)
    idx = np.argmin(np.abs(far - frr))
    return far[idx] * 100  # EER in percent

Common pitfalls

• EER% is highly sensitive to the decision threshold; systems must report the exact operating point or use min-CER/HTTER conventions for fair comparison.
• Cross-dataset evaluation (e.g., training on ASVspoof 2019 and testing on 2021/2024 or EmoFake) often reveals severe domain shift; models may overfit to specific TTS/VC systems used in training.
• Prosodic features (F0, voiced/unvoiced) are critical for emotional spoofing detection; ignoring them leads to high error rates on EmoFake and Mixed Emotions.

Evidence (verbatim from paper)

We evaluate HuLA across a diverse set of corpora, using two datasets for training and the rest only for evaluation. This design ensures that performance is assessed under realistic, out-of-domain conditions. TABLE IV: Comparison of baseline and proposed models on the ASVspoof evaluation sets in terms of EER%.

Citation

@misc{mahapatra2025hula,
  title={HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech},
  author={Mahapatra et al. (2025)},
  year={2025},
  note={arXiv:2509.21676}
}

• arXiv: 2509.21676