qhjqhj00 human-fooling-rate
The State Of TTS: A Case Study with Human Fooling Rates — Varadhan et al. (2025) (arXiv:2508.04179, 2025)
What this evaluates
Probes whether text-to-speech systems can perceptually deceive human listeners into believing synthetic speech is real. It measures the gap between traditional preference scores (CMOS/MUSHRA) and actual indistinguishability, highlighting how prompt expressivity and model type affect deception capability.
Datasets
- LJSpeech — total ?; splits: test (-1)
- LibriTTS — total ?; splits: test (-1)
- LibriSpeech — total ?; splits: test (-1)
- Expresso — total ?; splits: test (-1)
Metrics
Human Fooling Rate (HFR)(primary) — range: percent- Percentage of synthetic speech samples that human listeners incorrectly identify as real human speech. Calculated as (number of samples rated as real / total samples rated) × 100.
Input / output format
Input: Audio samples of synthetic speech generated by TTS models, presented through the SAFFRON platform. Listeners hear full samples under controlled conditions without rushing.
Output: Binary judgment indicating whether the listener perceives the audio as real human speech or synthetic.
Scoring recipe
def compute_hfr(listener_judgments):
# listener_judgments: list of 'real' or 'synthetic' per sample
fooled = sum(1 for j in listener_judgments if j == 'real')
total = len(listener_judgments)
return (fooled / total) * 100 if total > 0 else 0.0
Common pitfalls
- Using low-expressivity or monotonic reference prompts (e.g., standard LJSpeech test sets) artificially inflates HFR by making synthesis easier, masking true realism gaps.
- Rushed judgments or partial listening bypasses platform controls, leading to unreliable fooling rates.
- Confusing HFR with preference metrics like CMOS or MUSHRA; high preference scores do not guarantee high deception rates.
Evidence (verbatim from paper)
SAFFRON supports both HFR and MUSHRA tests, ensuring a standardized and reproducible framework for benchmarking speech realism. SAFFRON enforces strict listening conditions by requiring participants to hear full samples before responding, tracking response times to prevent rushed judgments, and integrating seamlessly with Prolific for large-scale crowd-sourced evaluations.
Citation
@misc{varadhan2025stateoftts,
title={The State Of TTS: A Case Study with Human Fooling Rates},
author={Varadhan et al. (2025)},
year={2025},
note={arXiv:2508.04179}
}
- arXiv: 2508.04179