Voice Cloning Accent Eval

voice-cloning-accent-eval

Acoustic and perceptual differences between standard and accented Chinese speech and their voice clones — Tianle Yang et al. (2026) (arXiv:2604.01562, 2026)

What this evaluates

Evaluates how commercial voice cloning systems preserve speaker identity and speech intelligibility for standard versus accented Mandarin speakers. It probes the alignment between acoustic embedding distances and human perceptual judgments of similarity and intelligibility across accent conditions.

Datasets

• Standard and Accented Mandarin Speech Dataset — total ?; splits: standard (-1), accented (-1)

Metrics

• ECAPA-TDNN cosine distance — range: other
  • Cosine distance between speaker embeddings extracted by the ECAPA-TDNN model. Computed as mean distance between original and clone ($\overline{d}{OC}$) and within-original pairs ($\overline{d}{OO}$).
• Speaker similarity rating — range: other
  • Human listener ordinal rating on a 1–5 scale indicating how similar a voice clone sounds to its original recording.
• Intelligibility gain score (primary) — range: other
  • Within-listener baseline-corrected intelligibility improvement: $\Delta_{\mathrm{intell}} = \overline{r}{\mathrm{clone}} - \overline{r}{\mathrm{orig}}$, where $\overline{r}$ is the mean 1–5 intelligibility rating for the clone and original respectively.

Input / output format

Input: Speech recordings of Mandarin speakers (standard and accented) processed by commercial voice cloning APIs (ElevenLabs, AnyVoice, MiniMax) to generate clones. For perceptual evaluation, listeners receive paired original and cloned audio stimuli.

Output: Acoustic evaluation outputs mean cosine distances per speaker. Perceptual evaluation outputs 1–5 ordinal ratings for similarity and intelligibility, which are then averaged and differenced to compute gain scores.

Scoring recipe

def compute_clone_divergence(originals, clones, embed_fn):
    d_oc = mean(cosine_distance(embed_fn(o), embed_fn(c)) for o, c in zip(originals, clones))
    d_oo = mean(cosine_distance(embed_fn(o1), embed_fn(o2)) for o1, o2 in combinations(originals, 2))
    return d_oc - d_oo

def compute_intelligibility_gain(listener_ratings_orig, listener_ratings_clone):
    r_orig = mean(listener_ratings_orig)
    r_clone = mean(listener_ratings_clone)
    return r_clone - r_orig

Common pitfalls

• Treating token-pair distances as independent observations instead of averaging per speaker, which causes pseudo-replication.
• Assuming acoustic embedding divergence directly correlates with perceptual similarity, as the study found they diverge (clones rated less similar for accented speech despite similar embedding distances).

Evidence (verbatim from paper)

For intelligibility, we summarize each listener’s judgments using an intelligibility gain score that compares a clone directly to its matched original. For each participant, system, and speaker set (standard vs. accented), we first average the 1–5 intelligibility ratings across items separately for the original and the clone, and then compute gain as $\Delta_{\mathrm{intell}}=\overline{r}{\mathrm{clone}}-\overline{r}{\mathrm{orig}}$. Using gain has two advantages: it provides a within-listener, within-item baseline correction that reduces individual differences in rating scale use, and it targets the quantity of interest for voice cloning, namely how much intelligibility changes relative to the original rather than absolute ratings.

Citation

@misc{yang2026acoustic,
  title={Acoustic and perceptual differences between standard and accented Chinese speech and their voice clones},
  author={Tianle Yang et al. (2026)},
  year={2026},
  note={arXiv:2604.01562}
}

• arXiv: 2604.01562