Voice Indistinguishability Eval

voice-indistinguishability-eval

Voice-Indistinguishability: Protecting Voiceprint in Privacy-Preserving Speech Data Release — Han et al. (2020) (arXiv:2004.07442, 2020)

What this evaluates

Evaluates the privacy guarantee (voice-indistinguishability) and utility of perturbed speech data. It measures how effectively a sanitization framework hides speaker identity while preserving speech recognition accuracy and perceptual naturalness.

Datasets

• Librispeech (test-clean) — total ?; splits: test (-1)

Metrics

• ACC (Speaker Verification Accuracy) (primary) — range: [0, 1]
  • Accuracy of speaker verification using a PLDA model to compare x-vector embeddings of original versus modified speech utterances.
• CER (Character Error Rate) — range: [0, 1]
  • Character-level error rate calculated as (substitutions + deletions + insertions) / total characters in the ground truth transcript, measuring speech recognition utility.
• MOS (Mean Opinion Score) — range: other
  • Average rating from 15 human listeners on a Likert scale, assessing dissimilarity (privacy) and naturalness (utility) of modified voices.

Input / output format

Input: Original speech utterances (waveforms) from the Librispeech test-clean set.

Output: Modified/sanitized speech utterances generated by the proposed perturbation framework.

Scoring recipe

def evaluate(audio_orig, audio_mod, transcript):
    # ACC: Speaker Verification
    x_orig = extract_xvector(audio_orig)
    x_mod = extract_xvector(audio_mod)
    score = plda_similarity(x_orig, x_mod)
    acc = threshold_decision(score, threshold)

    # CER: Speech Recognition
    pred = deepspeech_recognize(audio_mod)
    cer = edit_distance(pred, transcript) / len(transcript)

    # MOS: Subjective
    mos = mean([listener_rating(audio_orig, audio_mod) for _ in range(15)])
    return acc, cer, mos

Common pitfalls

• Confusing the privacy budget epsilon with utility: smaller epsilon increases privacy but may not linearly decrease utility due to the mechanism's randomness.
• Using different datasets for x-vector pre-training (VoxCeleb/SITW) vs evaluation (Librispeech) without accounting for potential domain shift in verification scores.
• MOS relies on a small human panel (15 listeners) and subjective scales, making cross-study comparison difficult without standardized protocols.

Evidence (verbatim from paper)

We evaluate the effect of the size of the speech database and the privacy budget $\epsilon$ on the changes between the original voices and the modified voices. The Mean Squared Error (MSE) and the accuracy of speaker verification through PLDA (ACC) are used. For testing, the character level accuracy is calculated. We use the character error rate (CER) to evaluate the performance of speech recognition for protected speech data after perturbing the voiceprint. The MOS (Mean Opinion Score) is used to evaluate the generated sound.

Citation

@misc{han2020voiceindistinguishability,
  title={Voice-Indistinguishability: Protecting Voiceprint in Privacy-Preserving Speech Data Release},
  author={Han et al. (2020)},
  year={2020},
  note={arXiv:2004.07442}
}

• arXiv: 2004.07442