Nvtts Eval

nvtts-eval

NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech — Borisov et al. (2025) (arXiv:2507.13155, 2025)

What this evaluates

This evaluation protocol assesses the capability of zero-shot text-to-speech models to synthesize nonverbal vocalizations (NVs) like breathing, laughter, coughing, and sighs alongside emotional speech. It measures speech intelligibility, speaker and emotion fidelity, acoustic quality, and the precise alignment of generated NVs with reference audio.

Datasets

• NVTTS — total ?; splits: train (3820), dev (46), test (366)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed by transcribing generated audio using Whisper-large-v3-turbo and comparing against the ground-truth text transcript.
• SIM-o — range: [0, 1]
  • Speaker similarity score computed using wavlm-base-plus-sv embeddings to measure cosine similarity between reference and generated speaker embeddings.
• Emo-sim — range: [0, 1]
  • Emotion similarity score measuring the alignment of emotional content between reference and generated audio.
• NV Jaccard distance — range: [0, 1]
  • Jaccard distance between the set of detected nonverbal vocalizations in the reference audio and the generated audio, computed using the BEATs model. Separate scores are reported for breathing, laughter, and coughing.
• DNSMOS — range: other
  • Deep Noise Suppression Mean Opinion Score, an automatic objective metric for predicting human-perceived audio quality and noise robustness.
• Preference — range: percent
  • Human side-by-side preference rate where annotators choose the better model based solely on NV-containing transcriptions without reference audio.

Input / output format

Input: Text prompt containing spoken words, nonverbal vocalization tags, and emotion labels, paired with a reference audio clip for zero-shot voice cloning.

Output: Synthesized audio waveform matching the input text, NV tags, and emotion.

Scoring recipe

def compute_metrics(predictions, gold):
    scores = {'WER': 0, 'SIM-o': 0, 'NV_Jaccard': 0, 'DNSMOS': 0}
    for pred_audio, ref_audio, gold_text, nv_tags in zip(predictions, gold['audio'], gold['text'], gold['nv']):
        pred_text = whisper.transcribe(pred_audio, model='large-v3-turbo')
        scores['WER'] += wer_score(pred_text, gold_text)
        ref_emb = wavlm_base_plus_sv.encode(ref_audio)
        pred_emb = wavlm_base_plus_sv.encode(pred_audio)
        scores['SIM-o'] += cosine_similarity(ref_emb, pred_emb)
        ref_nvs = beat.detect_nv(ref_audio)
        pred_nvs = beat.detect_nv(pred_audio)
        scores['NV_Jaccard'] += jaccard_distance(ref_nvs, pred_nvs)
        scores['DNSMOS'] += dnsmos.predict(pred_audio)
    n = len(predictions)
    return {k: v/n for k, v in scores.items()}

Common pitfalls

• NV Jaccard distance is highly dependent on the BEATs detection model and thresholds; results are not directly comparable across different NV detection pipelines.
• The test set is heavily skewed toward Neutral and Happy emotions, so emotion generation capabilities are only reliably assessed for the Happy category.
• Human preference tests excluded reference audio to isolate NV effects, which may overestimate the importance of NV alignment relative to natural voice quality in real-world usage.

Evidence (verbatim from paper)

We evaluate model performance on both subjective and objective metrics, including word error rate (WER), speaker similarity (SIM-o) [29], emotion similarity (Emo-sim) [22], NV Jaccard distance [19], DNSMOS [30], and preference test (Preference) on the test set, which includes 359 samples.

Citation

@misc{borisov2025nonverbaltts,
  title={NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech},
  author={Borisov et al. (2025)},
  year={2025},
  note={arXiv:2507.13155}
}

• arXiv: 2507.13155