Psp Accent Eval

psp-accent-eval

PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech — Menta et al. (2026) (arXiv:2604.25476, 2026)

What this evaluates

Evaluates the phonological accent fidelity and prosodic naturalness of Indic text-to-speech systems across Hindi, Telugu, and Tamil. It decomposes accent into per-phoneme dimensions (retroflex, aspiration, Tamil-zha, vowel-length) and corpus-level distributional metrics, revealing gaps between intelligibility and native-like accent.

Datasets

• PSP Benchmark Sets — total 300; splits: pilot (10), golden (300); repo https://github.com/praxelhq/praxy

Metrics

• FAD (primary) — range: other
  • Fréchet Audio Distance computed between Wav2Vec2-XLS-R phonetic embeddings of generated audio and a native reference distribution (1000 utterances). Lower values indicate closer alignment to native acoustic distributions.
• PSD — range: other
  • Prosodic Signature Divergence measuring distance in a 5D prosodic space (pitch range, log-F0, speech rate, nPVI, log-duration) from native speaker centroids. Lower values indicate more natural prosody.
• Retroflex/Aspiration/Tamil-zha/Length Fidelity — range: percent
  • Per-token collapse rates calculated via forced alignment, measuring the percentage of phonological tokens correctly produced versus collapsed. Expressed as a percentage.
• LLM-WER — range: percent
  • Word error rate computed by a Qwen-2.5-72B semantic judge over Whisper ASR transcripts to measure semantic intelligibility.

Input / output format

Input: Text prompts in Hindi, Telugu, or Tamil, stratified by phonological density (retroflex-heavy, aspiration-heavy, length-heavy, conjunct-heavy, general). Commercial systems also receive voice gender prompts.

Output: Synthesized audio files (WAV) for each text prompt.

Scoring recipe

def score_system(generated_audios, native_reference_audios, ground_truth_texts):
    # 1. Per-token collapse rates via forced alignment
    aligned = forced_align(generated_audios)
    collapse_rate = sum(1 for t in aligned if t.feature == 'collapsed') / len(aligned)
    
    # 2. FAD using Wav2Vec2-XLS-R embeddings
    gen_emb = wav2vec2_xls_r.encode(generated_audios)
    ref_emb = wav2vec2_xls_r.encode(native_reference_audios)  # 1000 utts
    FAD = frechet_distance(gen_emb.mean(), gen_emb.cov(), ref_emb.mean(), ref_emb.cov())
    
    # 3. PSD from prosodic features
    prosodic_vec = extract_prosody(generated_audios)  # [pitch_range, log_F0, rate, nPVI, log_dur]
    PSD = cosine_distance(prosodic_vec, native_prosodic_centroid)
    
    # 4. LLM-WER
    transcript = whisper.transcribe(generated_audios)
    LLM_WER = qwen2_5_72b.compute_wer(transcript, ground_truth_texts)
    return collapse_rate, FAD, PSD, LLM_WER

Common pitfalls

• WER and FAD/PSD orderings are often inversely correlated; a system with the lowest WER may rank last on accent metrics.
• Pilot sets have sample-size asymmetry (commercial systems use 2 voice genders/20 wavs, open-source use 1 voice/10 wavs), which can skew preliminary collapse rates.
• FAD and PSD are corpus-level metrics requiring large native reference distributions (1000 and ~500 utterances), not per-utterance scores.
• Improving spectral distance (FAD) can sometimes regress prosodic naturalness (PSD) if only acoustic weights are updated without prosodic conditioning.

Evidence (verbatim from paper)

Each utterance is scored on all applicable per-phoneme PSP dimensions; corpus-level FAD and PSD are computed once per (system, language) against native reference distributions of 1000 and 500 utterances respectively.

Citation

@misc{menta2026psp,
  title={PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech},
  author={Menta et al. (2026)},
  year={2026},
  note={arXiv:2604.25476}
}

• arXiv: 2604.25476