Librispeech Edit Eval

librispeech-edit-eval

AST: Adaptive, Seamless, and Training-Free Precise Speech Editing — Lv et al. (2026) (arXiv:2604.16056, 2026)

What this evaluates

Evaluates speech editing models on their ability to accurately modify target words while preserving speaker identity, acoustic quality, and temporal alignment of unedited regions.

Datasets

• LibriSpeech-Edit — total 2000; splits: test (2000)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: percentage of incorrectly recognized words relative to the reference transcript, computed using Whisper large-v3 transcriptions of the generated audio.
• DNSMOS — range: other
  • DNSMOS: Neural network-based proxy for Mean Opinion Score (MOS) following ITU-T P.808, assessing overall audio quality and naturalness.
• SpkSim — range: [0, 1]
  • Speaker Similarity: cosine similarity between speaker embeddings extracted from the source and edited speech using WavLM.
• WDTW — range: other
  • Word-level Dynamic Time Warping: length-normalized DTW distance between word-level temporal segments of the original and edited speech, computed via forced alignment.

Input / output format

Input: Source audio waveform, source transcript, and target/edit transcript.

Output: Edited audio waveform.

Scoring recipe

pred_transcript = whisper.transcribe(edited_audio, model='large-v3')
wer = 100 * edit_distance(pred_transcript, target_transcript) / len(target_transcript)
dnsmos = dns_mos_model.predict(edited_audio)
src_emb = wavlm.encode(source_audio)
edit_emb = wavlm.encode(edited_audio)
spksim = cosine_similarity(src_emb, edit_emb)
t_src = force_align(source_audio, source_transcript)
t_edit = force_align(edited_audio, target_transcript)
seg_src = extract_word_segments(t_src)
seg_edit = extract_word_segments(t_edit)
wdtw = dtw(seg_src, seg_edit) / total_duration

Common pitfalls

• WER is computed on Whisper-generated transcriptions of the model output, not directly on the raw audio against ground truth.
• WDTW specifically measures temporal fidelity of unedited regions by aligning word-level segments, not global utterance alignment.
• DNSMOS is a learned proxy for human MOS ratings and may not perfectly correlate with subjective quality across all acoustic conditions.

Evidence (verbatim from paper)

To comprehensively evaluate the quality of the edited speech, we employ four objective metrics covering textual accuracy, acoustic quality, speaker preservation, and temporal consistency: Word Error Rate (WER): Evaluates the intelligibility and textual accuracy of the generated speech. DNSMOS: A robust neural network-based metric serving as a proxy for the standardized Mean Opinion Score (MOS) following the ITU-T P.808 recommendation, utilized here to assess overall audio quality and naturalness. Speaker Similarity (SpkSim): Quantifies the preservation of speaker identity. We extract speaker embeddings utilizing WavLM and compute the cosine similarity between the embeddings of the source and edited speech. Word-level Dynamic Time Warping (WDTW): A novel metric proposed in this work to measure the temporal fidelity of the unedited regions.

Citation

@misc{lv2026ast,
  title={AST: Adaptive, Seamless, and Training-Free Precise Speech Editing},
  author={Lv et al. (2026)},
  year={2026},
  note={arXiv:2604.16056}
}

• arXiv: 2604.16056