brycewang-stanford INTERSPEECH Reproducibility
Speech results decay through the measurement, not just the model. Two labs with identical checkpoints can report WERs a point apart because their text normalization differs, and two identical TTS systems can score half a MOS point apart under different listening panels. Interspeech reviewers know this, and the 4-page format means reproducibility is asserted through precise, compressed disclosure — there is no appendix to hide vagueness in.
The five decay channels
| Channel | How results drift | Pin it by |
|---|---|---|
| Corpus version & splits | "test" ≠ official test; filtered utterances | Name corpus version + official partition; list any filtering rule |
| Text normalization | casing, punctuation, numerals change WER | Publish the norm script; name the scorer (e.g., sclite/jiwer + config) |
| Trial lists / protocols | EER/minDCF move with the trial set | Cite the exact trial list file and calibration set |
| Subjective testing | MOS panels differ in raters, scale, stimuli | Report rater count, platform, instructions, #stimuli, CI |
| Training stochasticity | seed, data order, nondeterministic kernels | Seeds logged; ≥3 runs where feasible; mean ± sd |
Corpus discipline
- Name the release, not the family: "LibriSpeech test-other", "VoxCeleb1-O cleaned trial list", "CHiME-6 eval per the challenge rules" — each token is checkable.
- If you re-partition, justify it and publish the split manifests; private splits are the single most common irreproducibility at this venue.
- State licenses (see
interspeech-artifact-evaluation); a reader must know whether they can obtain your training data. - For multilingual claims, list the languages and hours per language — "50+ languages" without a table row per language is not reproducible.
Objective metrics: publish the ruler
WER is a Levenshtein distance over a normalization you chose. Reviewers who have been burned will ask:
- Which scoring tool and version, with which options?
- What text normalization (case, punctuation, number expansion, compound rules)?
- Insertion-penalty / LM-weight tuning: on which dev set?
- For EER/minDCF: which trial list, which prior/costs in the DCF?
- For enhancement: PESQ-WB or NB? STOI or ESTOI? Which reference alignment?
One sentence in the paper plus scripts in the repo answers all of it.
Subjective metrics: the protocol is the result
A MOS number without its protocol is decoration. The reportable minimum, fitted to about three lines of a 4-page paper:
Naturalness MOS: 5-point ACR; N=30 crowd raters (platform X), native speakers,
screened by anchor trials; 20 utterances/system, 8 systems, randomized;
95% CI via rater bootstrap. CMOS vs. baseline on the same panel.
If a claim rests on ±0.1 MOS, it rests on nothing — pair MOS with CMOS or an objective proxy (e.g., a learned MOS predictor, clearly labeled as a proxy) and say whether the panel can resolve the difference.
Variance and seeds inside 4 pages
- Prefer one honest line — "mean ± sd over 3 seeds; CI excludes the baseline" — over three extra ablations without variance.
- Bootstrap CIs at the utterance level for WER; matched-pairs significance (e.g., MAPSSWE-style or paired bootstrap) when claiming a system beats another.
- If compute allows only one run, say so and temper the claim's verbs accordingly.
Recipe rerunnability
Most Interspeech systems live in community toolkits (ESPnet, SpeechBrain, Kaldi, NeMo, k2, fairseq lineage). Reproducibility there means:
- Pin the toolkit commit and the recipe directory, not just the toolkit name.
- Diff-against-default: state which hyperparameters differ from the stock recipe — that diff is your method's footprint.
- Log the full config used for each table row; map rows to config files in the repo.
- Record hardware and wall-clock: "8×A100, 36h" tells readers whether they can replicate at all.
Review-time vs camera-ready disclosure
During double-blind review the repo must be anonymized or described-but-withheld; at camera-ready the links flip public. Write the reproducibility sentences so that only the URL changes, not the promises.
The disclosure block, worth three lines of the paper
A compact pattern that closes most reviewer doubts at once — adapt per task:
Setup: ESPnet2 (commit abc123), LibriSpeech 960h official splits;
Whisper-normalizer text rules; WER via jiwer 3.x, config in repo.
Training: 3 seeds (mean±sd reported), 8×A100, 30h/run; decoding:
beam 10, no external LM; dev-clean used for all tuning.
Every clause preempts a specific review question; nothing in it costs a figure.
Reviewer questions to preempt
- "Is the gain inside seed noise?" → variance line present.
- "Which normalization produced these WERs?" → ruler named and shipped.
- "Did any tuning touch the test set?" → dev-only sentence present.
- "Can I obtain the training data?" → corpus + license named.
- "Would the MOS panel resolve this difference?" → CI + panel size reported.
If the paper answers all five before they are asked, the reproducibility paragraph has done its rhetorical job as well as its scientific one.
Output format
[Decay-channel audit] corpus / normalization / trials / subjective / seeds — pinned?
[Ruler published?] scoring tools + norm rules in repo: yes / partial / no
[Subjective protocol] complete per the reportable minimum? gaps
[Variance] runs, CI method, significance test used
[Recipe] toolkit+commit, config-per-row mapping, hardware line
[Weakest link] <the one channel most likely to break replication>
Cross-check any cycle-specific checklist or disclosure field on the live CFP —
Interspeech has been adding evaluation-rigor language cycle by cycle (sources:
resources/official-source-map.md, checked 2026-07-08).