Revise Eval

revise-eval

ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement — Hsu et al. (2022) (arXiv:2212.11377, 2022)

What this evaluates

Evaluates generalized speech enhancement and audio-visual speech resynthesis across multiple distortion types (denoising, separation, inpainting, video-to-speech). Measures content intelligibility, audio-video synchronization, perceptual quality, and low-level signal reconstruction.

Datasets

• LRS3 — total ?; splits: test (-1)
• EasyCom — total ?; splits: train (-1), valid (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: percentage of incorrectly recognized words relative to the reference transcript, computed using a public ASR model.
• LSE-D — range: other
  • SyncNet temporal distance: average predicted temporal distance between audio and video frames over the test set.
• LSE-C — range: other
  • SyncNet confidence: average confidence score of the audio-video synchronization prediction over the test set.
• MOS — range: [1, 5]
  • Mean Opinion Score: subjective quality rating on a 1–5 scale (0.5 increments), averaged across ≥15 raters per sample using CrowdMOS.
• ESTOI — range: [0, 1]
  • Extended Short-Time Objective Intelligibility: estimates speech intelligibility based on temporal envelope correlation between reference and enhanced speech.
• MCD — range: other
  • Mel Cepstral Distortion: measures the Euclidean distance between mel cepstral coefficients of reference and synthesized speech.

Input / output format

Input: Video frames (head-crop at 25fps) and/or audio features (23-dim Mel FBank, 10ms shift, stacked every 4 frames). For EasyCom, beamformed multi-channel audio or single-channel distant mic audio paired with video.

Output: Resynthesized audio waveform (16kHz sampling rate).

Scoring recipe

def compute_metrics(predictions, references, videos=None):
    wer = compute_wer(predictions, references, asr_model='public_model')
    lse_d, lse_c = compute_syncnet(predictions, videos)
    mos = compute_mos(predictions, num_raters=15, scale=[1,5])
    estoi = compute_estoi(predictions, references)
    mcd = compute_mcd(predictions, references)
    return {'WER': wer, 'LSE-D': lse_d, 'LSE-C': lse_c, 'MOS': mos, 'ESTOI': estoi, 'MCD': mcd}

Common pitfalls

• Using the distant microphone audio as the clean reference for EasyCom instead of the close-talking microphone.
• Assuming LRS3 contains real-world noisy speech; all enhancement tasks are artificially corrupted (mixed with AudioSet noise, other speakers, or zeroed frames).
• MOS evaluation requires exactly 50 randomly sampled test files and ≥15 raters per file via CrowdMOS, not ad-hoc listening.

Evidence (verbatim from paper)

For content, we use the WER computed with a speech recognition model to measure the intelligibility quantitatively similar to[[36]], [[37]]. The public model from [[56]] is used, which reports a WER of 5.6% on LRS3 test split and 35.7% on the EasyCom close-talking validation set. For synchronization, following[[21]] we use SyncNet[[9]] metrics, the predicted temporal distance between audio and video (LSE-D) and the prediction’s confidence (LSE-C) are averaged over the entire test set. For quality, we follow the tradition of text-to-speech synthesis evaluation and conduct subjective mean opinion score (MOS) studies with a scale from 1 to 5 and a 0.5 increment. We evaluate 50 randomly sampled files from the test set, where each sample was evaluated by at least 15 raters using the CrowdMOS package[[47]]. Finally, to evaluate reconstruction of low-level details as typically done in speech denoising or source separation studies, we include ESTOI[[51]] and Mel cepstral distortion (MCD)[[28]].

Citation

@misc{hsu2022revise,
  title={ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement},
  author={Hsu et al. (2022)},
  year={2022},
  note={arXiv:2212.11377}
}

• arXiv: 2212.11377