Timit Tts Eval

timit-tts-eval

TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection — Salvi et al. (2022) (arXiv:2209.08000, 2022)

What this evaluates

Evaluates deepfake detectors on distinguishing real from synthetic audio and video, and on attributing synthetic audio to specific TTS generators. It probes robustness to post-processing (DTW alignment, augmentation) and video compression.

Datasets

• TIMIT-TTS — total ?; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• balanced accuracy — range: [0, 1]
  • Average of recall obtained on each class, calculated as the mean of per-class recall scores for multiclass tasks or (TP/(TP+FN) + TN/(TN+FP))/2 for binary tasks.

Input / output format

Input: Raw audio waveforms or video frames (optionally compressed with QP=23 or QP=40).

Output: Binary label (real/fake) for detection, or categorical label (TTS algorithm ID) for attribution.

Scoring recipe

def compute_auc(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_balanced_accuracy(y_true, y_pred):
    recalls = [recall_score(y_true == c, y_pred == c) for c in np.unique(y_true)]
    return np.mean(recalls)

Common pitfalls

• Models may learn speaker identity rather than TTS artifacts, inflating attribution scores for multi-speaker models.
• Performance drops significantly when testing on low-compression videos (QP=40) compared to high-quality training data, revealing robustness gaps.
• DTW post-processing lowers MOS and detection AUC, requiring explicit training on aligned tracks to avoid biased evaluation.

Evidence (verbatim from paper)

In particular, the operation that degrades the accuracy the most is the speech-to-speech alignment, with an AUC value that drops by $0.20$ between the clean and the DTW cases.

Citation

@misc{salvi2022timittts,
  title={TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection},
  author={Salvi et al. (2022)},
  year={2022},
  note={arXiv:2209.08000}
}

• arXiv: 2209.08000