Mscadd Eval

mscadd-eval

Multi-Speaker Conversational Audio Deepfake: Taxonomy, Dataset and Pilot Study — Alabi Ahmed et al. (2026) (arXiv:2602.00295, 2026)

What this evaluates

Evaluates audio deepfake detection models on their ability to distinguish real from synthetically generated multi-speaker conversations. It probes robustness to conversational dynamics, speech overlap, and varying acoustic conditions.

Datasets

• MsCADD — total 2830; splits: train (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for the fake class: 2 * (Precision * Recall) / (Precision + Recall).
• Accuracy — range: [0, 1]
  • Ratio of correctly classified instances (both real and fake) to the total number of instances.
• True Positive Rate (TPR) — range: [0, 1]
  • Recall for the fake class: TP / (TP + FN).
• True Negative Rate (TNR) — range: [0, 1]
  • Specificity for the real class: TN / (TN + FP).

Input / output format

Input: Audio clips containing two-speaker conversations (clean or noisy conditions).

Output: Binary classification label (real vs. fake) or aggregated per-clip fake probability scores.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * precision * tpr / (precision + tpr) if (precision + tpr) > 0 else 0
    return {'accuracy': accuracy, 'tpr': tpr, 'tnr': tnr, 'f1': f1}

Common pitfalls

• Decision thresholds are empirically tuned per model/dataset rather than fixed (e.g., -15.82 for LFCC-LCNN).
• Models exhibit high false positive rates on real conversational speech due to acoustic complexity and overlap.
• Pilot study does not report performance breakdowns by TTS system or speaker gender composition.

Evidence (verbatim from paper)

We report accuracy (percentage of correctly classified instances), true positive rate (TPR), True negative rate (TNR) and F1 score for fake class.

Citation

@misc{alabi2026multispeaker,
  title={Multi-Speaker Conversational Audio Deepfake: Taxonomy, Dataset and Pilot Study},
  author={Alabi Ahmed et al. (2026)},
  year={2026},
  note={arXiv:2602.00295}
}

• arXiv: 2602.00295