Audio Turing Test Eval

audio-turing-test-eval

Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese — Wang et al. (2025) (arXiv:2505.11200, 2025)

What this evaluates

Evaluates the human-likeness of Chinese text-to-speech systems using a Turing-test-inspired protocol where human listeners classify audio as human, unclear, or machine. It also benchmarks an automatic LLM-based evaluator against human judgments and traditional MOS prediction models to measure alignment and trap-item detection capability.

Datasets

• ATT-Corpus — total ?; splits: test (-1)

Metrics

• HLS (primary) — range: [0, 1]
  • Human-likeness Score (HLS) is the mean of human ratings per clip, where [Human]=1.0, [Unclear]=0.5, and [Machine]=0.0.
• F1 score — range: [0, 1]
  • Binary classification F1 on trap items (human vs synthetic) using a 0.5 decision threshold on predicted scores.
• Kendall τ distance — range: [0, 1]
  • Rank correlation distance between predicted HLS and human evaluation rankings; lower values indicate better alignment.

Input / output format

Input: Audio clips (synthesized TTS output or human speech) presented to human evaluators or processed by Auto-ATT/Qwen2-Audio-Instruct.

Output: Human: single-choice label ([Human], [Unclear], [Machine]) plus written justification. Auto-ATT: continuous HLS prediction score.

Scoring recipe

def compute_hls(human_choices):
    ratings = [1.0 if c == 'Human' else 0.5 if c == 'Unclear' else 0.0 for c in human_choices]
    return sum(ratings) / len(ratings)

def compute_f1_auto_att(auto_scores, gold_labels):
    preds = [1 if s >= 0.5 else 0 for s in auto_scores]
    return f1_score(gold_labels, preds)

def compute_kendall_tau(pred_rankings, human_rankings):
    return kendalltau(pred_rankings, human_rankings).correlation

Common pitfalls

• Confusing HLS with traditional Mean Opinion Score (MOS); HLS is more sensitive to subtle synthetic artifacts and reveals capability gaps MOS misses.
• Misinterpreting Kendall τ direction; the paper reports it as a 'distance' where lower values indicate better alignment, unlike standard correlation where higher is better.
• Assuming black-box vs white-box evaluation settings introduce systematic bias; the paper shows performance gaps are uniform across both.

Evidence (verbatim from paper)

Auto-ATT vastly outperformed the baselines, achieving an F1 score of 0.92, while UTMOSv2 reached only 0.14 and DNSMOSPro collapsed to 0.00 at the 0.5 decision threshold.

Citation

@misc{wang2025audioturingtest,
  title={Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.11200}
}

• arXiv: 2505.11200