Synparaspeech Eval

synparaspeech-eval

SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding — Bai et al. (2025) (arXiv:2509.14946, 2025)

What this evaluates

Evaluates the effectiveness of an automated framework for synthesizing paralinguistic speech datasets on downstream paralinguistic text-to-speech generation and event detection tasks. It measures how well the generated data improves model performance in producing and recognizing paralinguistic features like laughter, sighs, and gasps compared to real-world annotated datasets.

Datasets

• SynParaSpeech — total ?; splits: train (-1), val (-1); repo https://github.com/ShawnPi233/SynParaSpeech

Metrics

• PMOS (primary) — range: other
  • Paralinguistic MOS: a 5-point subjective rating scale used to evaluate the quality of synthesized paralinguistic features.
• Accuracy — range: [0, 1]
  • The proportion of correctly predicted paralinguistic event labels out of the total test instances.
• Macro F1 Score — range: [0, 1]
  • The unweighted mean of recall or precision computed independently for each paralinguistic class, then averaged.
• CER — range: percent
  • Character Error Rate: the edit distance between the predicted and reference transcriptions normalized by the reference length, measuring intelligibility.
• SECS — range: other
  • Speaker Similarity Score: an objective metric quantifying the acoustic similarity between the synthesized and reference speaker embeddings.
• UTMOSv2 — range: other
  • An objective speech quality score predicted by a dedicated model, calibrated to align with human listening tests.
• NMOS — range: other
  • Naturalness MOS: a 5-point subjective rating for the overall naturalness of the synthesized audio.
• SMOS — range: other
  • Speaker Similarity MOS: a 5-point subjective rating for how well the synthesized voice matches the target speaker.
• QMOS — range: other
  • Overall Quality MOS: a 5-point subjective rating for the combined audio quality of the synthesized output.

Input / output format

Input: For TTS: text prompts with inserted paralinguistic labels (e.g., [laugh], [sigh]). For Event Detection: audio clips paired with varying numbers of context prompts (1 to 7-shot).

Output: For TTS: synthesized audio waveforms. For Event Detection: predicted paralinguistic event category labels.

Scoring recipe

def score_tts(predictions, references):
    cer = edit_distance(predictions, references) / len(references)
    mos_scores = collect_5point_ratings(predictions) # PMOS, NMOS, SMOS, QMOS
    return cer, mos_scores

def score_detection(predictions, labels):
    accuracy = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    f1_macro = mean(f1_score(y_true=labels, y_pred=predictions, average='macro'))
    return accuracy, f1_macro

Common pitfalls

• CER may artificially increase for paralinguistic speech because sounds like laughter are transcribed literally (e.g., 'ha ha'), penalizing intelligibility metrics designed for standard speech.
• Objective metrics like CER, SECS, and UTMOSv2 are calibrated for standard speech and may show slight declines when evaluating paralinguistic synthesis, despite clear improvements in paralinguistic quality (PMOS).
• Context size in event detection has a non-monotonic effect; performance peaks at 5-shot and degrades at 7-shot due to input overload, requiring careful prompt selection.

Evidence (verbatim from paper)

Model performance was assessed using both objective and subjective metrics. Objective measures include character error rate (CER) for intelligibility, SECS for speaker similarity, and UTMOSv2 for speech quality. Subjective evaluation employed 5-point ratings: PMOS for the paralinguistic quality, NMOS for naturalness, SMOS for speaker similarity, and QMOS for overall audio quality. For MOS ratings, 21 volunteers participated in a double-blind evaluation. Performance was measured using accuracy, macro F1 score, and CER.

Citation

@misc{bai2025synparaspeech,
  title={SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding},
  author={Bai et al. (2025)},
  year={2025},
  note={arXiv:2509.14946}
}

• arXiv: 2509.14946