# Human Fooling Rate

> Probes whether text-to-speech systems can perceptually deceive human listeners into believing synthetic speech is real. It measures the gap between traditional preference scores (CMOS/MUSHRA) and actual indistinguishability, highlighting how prompt expressivity and model type affect deception capability. Use when the user has predictions and gold and needs to compute Human Fooling Rate (HFR).

- Skill: `qhjqhj00/human-fooling-rate` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/human-fooling-rate`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/human-fooling-rate/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/human-fooling-rate

---


# human-fooling-rate

> The State Of TTS: A Case Study with Human Fooling Rates — Varadhan et al. (2025) (arXiv:2508.04179, 2025)

## What this evaluates

Probes whether text-to-speech systems can perceptually deceive human listeners into believing synthetic speech is real. It measures the gap between traditional preference scores (CMOS/MUSHRA) and actual indistinguishability, highlighting how prompt expressivity and model type affect deception capability.

## Datasets

- **LJSpeech** — total ?; splits: test (-1)
- **LibriTTS** — total ?; splits: test (-1)
- **LibriSpeech** — total ?; splits: test (-1)
- **Expresso** — total ?; splits: test (-1)

## Metrics

- `Human Fooling Rate (HFR)` **(primary)** — range: percent
  - Percentage of synthetic speech samples that human listeners incorrectly identify as real human speech. Calculated as (number of samples rated as real / total samples rated) × 100.

## Input / output format

**Input**: Audio samples of synthetic speech generated by TTS models, presented through the SAFFRON platform. Listeners hear full samples under controlled conditions without rushing.

**Output**: Binary judgment indicating whether the listener perceives the audio as real human speech or synthetic.

## Scoring recipe

```python
def compute_hfr(listener_judgments):
    # listener_judgments: list of 'real' or 'synthetic' per sample
    fooled = sum(1 for j in listener_judgments if j == 'real')
    total = len(listener_judgments)
    return (fooled / total) * 100 if total > 0 else 0.0
```

## Common pitfalls

- Using low-expressivity or monotonic reference prompts (e.g., standard LJSpeech test sets) artificially inflates HFR by making synthesis easier, masking true realism gaps.
- Rushed judgments or partial listening bypasses platform controls, leading to unreliable fooling rates.
- Confusing HFR with preference metrics like CMOS or MUSHRA; high preference scores do not guarantee high deception rates.

## Evidence (verbatim from paper)

> SAFFRON supports both HFR and MUSHRA tests, ensuring a standardized and reproducible framework for benchmarking speech realism. SAFFRON enforces strict listening conditions by requiring participants to hear full samples before responding, tracking response times to prevent rushed judgments, and integrating seamlessly with Prolific for large-scale crowd-sourced evaluations.

## Citation

```bibtex
@misc{varadhan2025stateoftts,
  title={The State Of TTS: A Case Study with Human Fooling Rates},
  author={Varadhan et al. (2025)},
  year={2025},
  note={arXiv:2508.04179}
}
```

- arXiv: 2508.04179