Multiapi Spoof Eval

multiapi-spoof-eval

MultiAPI Spoof: A Multi-API Dataset and Local-Attention Network for Speech Anti-spoofing Detection — Zhang et al. (2025) (arXiv:2512.07352, 2025)

What this evaluates

Evaluates speech anti-spoofing detection and API source attribution capabilities on a large-scale dataset of synthetic speech generated by 30 distinct APIs. It probes model robustness to domain shifts, generalization to unseen spoofing sources, and fine-grained source identification in realistic, heterogeneous environments.

Datasets

• MultiAPI Spoof — total ?; splits: train (-1), dev (-1), eval (-1); repo https://github.com/XuepingZhang/MultiAPI-Spoof

Metrics

• EER (primary) — range: percent
  • Equal Error Rate: the operating threshold at which the false acceptance rate equals the false rejection rate. Lower is better.
• minDCF — range: percent
  • Minimum Decision Cost Function: the lowest weighted detection cost across all possible thresholds, calculated using specified prior probabilities and cost parameters. Lower is better.
• actDCF — range: percent
  • Actual Decision Cost Function: the weighted detection cost computed at a fixed operating point (typically the EER threshold) using specified priors and costs. Lower is better.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. For seen APIs, computed as the macro-average across 21 classes. For unseen APIs, computed on the single class. Overall performance is the macro-average across all classes including unseen.

Input / output format

Input: Normalized raw waveforms, uniformly segmented into 4-second clips (shorter clips are repeated, longer clips are truncated). No data augmentation is applied.

Output: Anti-spoofing: continuous score or probability indicating spoof likelihood. API Tracing: discrete class label prediction from 22 categories (21 seen APIs + 1 unseen class).

Scoring recipe

def score_antispoof(preds, labels):
    eer = compute_eer(preds, labels)  # threshold where FAR == FRR
    mindcf = compute_mindcf(preds, labels, prior=0.01, costs=[1, 10])
    actdcf = compute_actdcf(preds, labels, prior=0.01, costs=[1, 10])
    return {'EER': eer, 'minDCF': mindcf, 'actDCF': actdcf}

def score_apitrace(preds, labels):
    prec, rec, f1 = precision_recall_f1_score(labels, preds, average='macro')
    return {'precision': prec, 'recall': rec, 'F1': f1}

Common pitfalls

• Models trained exclusively on traditional public datasets (e.g., TIMIT, ASVspoof) exhibit severe domain shift and high EER on MultiAPI Spoof without additional domain-specific training data.
• API tracing suffers from low recall on unseen APIs despite high precision, as embeddings of unseen APIs mix with seen categories rather than forming separable clusters.

Evidence (verbatim from paper)

For the anti-spoofing task, performance is evaluated using Equal Error Rate (EER↓), minimum Decision Cost Function (minDCF↓), and actual Decision Cost Function (actDCF↓) [38]. For the API tracing task, we measure classification performance using precision, recall, and F1 [39]. Specifically, the F1 for seen APIs is computed as the macro-average of the F1 scores over the 21 seen APIs classes. For unseen APIs, F1 is computed on the single unseen class. The overall performance is reported as the macro-average across all classes, including the unseen class.

Citation

@misc{zhang2025multiapi,
  title={MultiAPI Spoof: A Multi-API Dataset and Local-Attention Network for Speech Anti-spoofing Detection},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2512.07352}
}

• arXiv: 2512.07352