Preset Voice Matching Eval

preset-voice-matching-eval

Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems — Platnick et al. (2024) (arXiv:2407.13153, 2024)

What this evaluates

Evaluates a privacy-regulated speech-to-speech translation framework that replaces voice cloning with matching to pre-consented preset voices. Probes classifier robustness, cross-lingual speech naturalness, and inference efficiency across multilingual scenarios.

Datasets

• RAVDESS — total ?; splits: train (-1), val (-1), test (-1)
• CGDD — total ?; splits: train (-1), val (-1), test (-1)
• CAFE — total ?; splits: (unstated)
• EmoDB — total ?; splits: (unstated)
• CREMA-D — total ?; splits: (unstated)

Metrics

• NISQA (primary) — range: [0, 5]
  • Non-intrusive Objective Speech Quality Assessment predicts speech quality scores without a reference signal, typically mapping to a [0, 5] scale similar to Mean Opinion Score (MOS).
• accuracy and precision — range: [0, 1]
  • Standard classification metrics computed per classifier (gender, male-emotion, female-emotion) to assess GEMO-Match feature extraction robustness.
• inference run-time — range: seconds
  • Average time taken to process 10 identical 15-second utterances per model, measured in seconds.

Input / output format

Input: Audio utterances for classifier evaluation; English text transcriptions and emotive English audio for multilingual S2ST testing; 15-second audio clips with transcriptions for run-time benchmarking.

Output: Predicted gender/emotion labels; target-language (French/German) synthesized speech audio; average inference time in seconds.

Scoring recipe

def compute_metrics(predictions, golds, audio_outputs, run_times):
    # Accuracy & Precision for classifiers
    accuracy = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    true_positives = sum(1 for p, g in zip(predictions, golds) if p == 1 and g == 1)
    predicted_positives = sum(1 for p in predictions if p == 1)
    precision = true_positives / max(predicted_positives, 1)
    
    # NISQA for speech naturalness
    nisqa_scores = [nisqa_model.predict(audio) for audio in audio_outputs]
    avg_nisqa = sum(nisqa_scores) / len(nisqa_scores)
    
    # Run-time
    avg_run_time = sum(run_times) / len(run_times)
    return {'accuracy': accuracy, 'precision': precision, 'nisqa': avg_nisqa, 'run_time': avg_run_time}

Common pitfalls

• The multilingual experiment bypasses the Speech-to-Text (STT) stage by feeding ground-truth English text directly to the translation model, which isolates GEMO-Match but ignores STT error propagation.
• NISQA provides objective quality predictions that may not fully capture perceptual differences in cross-lingual voice matching or emotional fidelity.
• Run-time comparisons use fixed 15-second utterances and may not reflect latency scaling for variable-length or highly dynamic speech.

Evidence (verbatim from paper)

We measured speaker naturalness using the standard metric Non-intrusive Objective Speech Quality Assessment (NISQA) [[42], [43]].

Citation

@misc{platnick2024preset,
  title={Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems},
  author={Platnick et al. (2024)},
  year={2024},
  note={arXiv:2407.13153}
}

• arXiv: 2407.13153