nvtts-eval
NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech — Borisov et al. (2025) (arXiv:2507.13155, 2025)
What this evaluates
This evaluation protocol assesses the capability of zero-shot text-to-speech models to synthesize nonverbal vocalizations (NVs) like breathing, laughter, coughing, and sighs alongside emotional speech. It measures speech intelligibility, speaker and emotion fidelity, acoustic quality, and the precise alignment of generated NVs with reference audio.
Datasets
- NVTTS — total ?; splits: train (3820), dev (46), test (366)
Metrics
WER (primary) — range: percent
- Word Error Rate computed by transcribing generated audio using Whisper-large-v3-turbo and comparing against the ground-truth text transcript.
SIM-o — range: [0, 1]
- Speaker similarity score computed using wavlm-base-plus-sv embeddings to measure cosine similarity between reference and generated speaker embeddings.
Emo-sim — range: [0, 1]
- Emotion similarity score measuring the alignment of emotional content between reference and generated audio.
NV Jaccard distance — range: [0, 1]
- Jaccard distance between the set of detected nonverbal vocalizations in the reference audio and the generated audio, computed using the BEATs model. Separate scores are reported for breathing, laughter, and coughing.
DNSMOS — range: other
- Deep Noise Suppression Mean Opinion Score, an automatic objective metric for predicting human-perceived audio quality and noise robustness.
Preference — range: percent
- Human side-by-side preference rate where annotators choose the better model based solely on NV-containing transcriptions without reference audio.
Input / output format
Input: Text prompt containing spoken words, nonverbal vocalization tags, and emotion labels, paired with a reference audio clip for zero-shot voice cloning.
Output: Synthesized audio waveform matching the input text, NV tags, and emotion.
Scoring recipe
def compute_metrics(predictions, gold):
scores = {'WER': 0, 'SIM-o': 0, 'NV_Jaccard': 0, 'DNSMOS': 0}
for pred_audio, ref_audio, gold_text, nv_tags in zip(predictions, gold['audio'], gold['text'], gold['nv']):
pred_text = whisper.transcribe(pred_audio, model='large-v3-turbo')
scores['WER'] += wer_score(pred_text, gold_text)
ref_emb = wavlm_base_plus_sv.encode(ref_audio)
pred_emb = wavlm_base_plus_sv.encode(pred_audio)
scores['SIM-o'] += cosine_similarity(ref_emb, pred_emb)
ref_nvs = beat.detect_nv(ref_audio)
pred_nvs = beat.detect_nv(pred_audio)
scores['NV_Jaccard'] += jaccard_distance(ref_nvs, pred_nvs)
scores['DNSMOS'] += dnsmos.predict(pred_audio)
n = len(predictions)
return {k: v/n for k, v in scores.items()}
Common pitfalls
- NV Jaccard distance is highly dependent on the BEATs detection model and thresholds; results are not directly comparable across different NV detection pipelines.
- The test set is heavily skewed toward Neutral and Happy emotions, so emotion generation capabilities are only reliably assessed for the Happy category.
- Human preference tests excluded reference audio to isolate NV effects, which may overestimate the importance of NV alignment relative to natural voice quality in real-world usage.
Evidence (verbatim from paper)
We evaluate model performance on both subjective and objective metrics, including word error rate (WER), speaker similarity (SIM-o) [29], emotion similarity (Emo-sim) [22], NV Jaccard distance [19], DNSMOS [30], and preference test (Preference) on the test set, which includes 359 samples.
Citation
@misc{borisov2025nonverbaltts,
title={NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech},
author={Borisov et al. (2025)},
year={2025},
note={arXiv:2507.13155}
}
1---2name: nvtts-eval3description: This evaluation protocol assesses the capability of zero-shot text-to-speech models to synthesize nonverbal vocalizations (NVs) like breathing, laughter, coughing, and sighs alongside emotional speech. It measures speech intelligibility, speaker and emotion fidelity, acoustic quality, and the precise alignment of generated NVs with reference audio. Use when the user wants to benchmark on NVTTS, or asks about evaluating this task. Reports WER.4metadata:5 source: https://skillmd.com/skills/qhjqhj00/nvtts-eval6 skill_kind: dataset_eval7 source_arxiv: 2507.131558 bibtex_key: borisov2025nonverbaltts9 confidence: high10---1112# nvtts-eval1314> NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech — Borisov et al. (2025) (arXiv:2507.13155, 2025)1516## What this evaluates1718This evaluation protocol assesses the capability of zero-shot text-to-speech models to synthesize nonverbal vocalizations (NVs) like breathing, laughter, coughing, and sighs alongside emotional speech. It measures speech intelligibility, speaker and emotion fidelity, acoustic quality, and the precise alignment of generated NVs with reference audio.1920## Datasets2122- **NVTTS** — total ?; splits: train (3820), dev (46), test (366)2324## Metrics2526- `WER` **(primary)** — range: percent27 - Word Error Rate computed by transcribing generated audio using Whisper-large-v3-turbo and comparing against the ground-truth text transcript.28- `SIM-o` — range: [0, 1]29 - Speaker similarity score computed using wavlm-base-plus-sv embeddings to measure cosine similarity between reference and generated speaker embeddings.30- `Emo-sim` — range: [0, 1]31 - Emotion similarity score measuring the alignment of emotional content between reference and generated audio.32- `NV Jaccard distance` — range: [0, 1]33 - Jaccard distance between the set of detected nonverbal vocalizations in the reference audio and the generated audio, computed using the BEATs model. Separate scores are reported for breathing, laughter, and coughing.34- `DNSMOS` — range: other35 - Deep Noise Suppression Mean Opinion Score, an automatic objective metric for predicting human-perceived audio quality and noise robustness.36- `Preference` — range: percent37 - Human side-by-side preference rate where annotators choose the better model based solely on NV-containing transcriptions without reference audio.3839## Input / output format4041**Input**: Text prompt containing spoken words, nonverbal vocalization tags, and emotion labels, paired with a reference audio clip for zero-shot voice cloning.4243**Output**: Synthesized audio waveform matching the input text, NV tags, and emotion.4445## Scoring recipe4647```python48def compute_metrics(predictions, gold):49 scores = {'WER': 0, 'SIM-o': 0, 'NV_Jaccard': 0, 'DNSMOS': 0}50 for pred_audio, ref_audio, gold_text, nv_tags in zip(predictions, gold['audio'], gold['text'], gold['nv']):51 pred_text = whisper.transcribe(pred_audio, model='large-v3-turbo')52 scores['WER'] += wer_score(pred_text, gold_text)53 ref_emb = wavlm_base_plus_sv.encode(ref_audio)54 pred_emb = wavlm_base_plus_sv.encode(pred_audio)55 scores['SIM-o'] += cosine_similarity(ref_emb, pred_emb)56 ref_nvs = beat.detect_nv(ref_audio)57 pred_nvs = beat.detect_nv(pred_audio)58 scores['NV_Jaccard'] += jaccard_distance(ref_nvs, pred_nvs)59 scores['DNSMOS'] += dnsmos.predict(pred_audio)60 n = len(predictions)61 return {k: v/n for k, v in scores.items()}62```6364## Common pitfalls6566- NV Jaccard distance is highly dependent on the BEATs detection model and thresholds; results are not directly comparable across different NV detection pipelines.67- The test set is heavily skewed toward Neutral and Happy emotions, so emotion generation capabilities are only reliably assessed for the Happy category.68- Human preference tests excluded reference audio to isolate NV effects, which may overestimate the importance of NV alignment relative to natural voice quality in real-world usage.6970## Evidence (verbatim from paper)7172> We evaluate model performance on both subjective and objective metrics, including word error rate (WER), speaker similarity (SIM-o) [29], emotion similarity (Emo-sim) [22], NV Jaccard distance [19], DNSMOS [30], and preference test (Preference) on the test set, which includes 359 samples.7374## Citation7576```bibtex77@misc{borisov2025nonverbaltts,78 title={NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech},79 author={Borisov et al. (2025)},80 year={2025},81 note={arXiv:2507.13155}82}83```8485- arXiv: 2507.13155