brycewang-stanford NAACL Reproducibility
Reproducibility at NAACL is enforced through a document, not a badge: the Responsible NLP checklist travels with the submission, reviewers read it against the paper, and answers contradicted by the PDF are grounds for rejection without review under current ARR policy. The working stance: every checklist answer is a claim you are prepared to defend in the author response.
The contract reading of the checklist
- Answer from what the paper contains, never from what the repo will eventually contain.
- "No" with a reason is a safe answer; an unsupported "Yes" is a trap you set for yourself.
- Point each answer at a location — section, table, appendix — so a reviewer verifying it lands somewhere specific.
- Recheck the answers after every major revision; checklists rot faster than papers.
What must be pinned for NLP results in 2026
| Moving part | Pin it as | Why it decays |
|---|---|---|
| Hosted LLM APIs | Model identifier + query date range | Providers swap weights behind stable names |
| Open-weights models | Exact checkpoint hash or revision tag | "Latest" changes under you |
| Decoding | Temperature, top-p, max tokens, seed policy, n samples | Unstated sampling makes numbers unrepeatable |
| Prompts | Verbatim strings, all variants, selection rule | "We used a standard prompt" reproduces nothing |
| Tokenizers / normalization | Version + Unicode normalization form | Silent retokenization shifts multilingual scores |
| Eval metrics | Implementation + version (not just the metric name) | Scorer variants disagree by whole points |
| Data splits | Published split files or generation script + seed | Ad-hoc splits are unrecoverable |
Multilingual runs: the NAACL-flavored decay modes
Papers committed to NAACL disproportionately evaluate across languages, and multilingual pipelines decay in language-specific ways: normalization that strips combining diacritics, sentence splitters that fail on Spanish inverted punctuation, tokenizers that fragment agglutinative morphology (Nahuatl, Quechua, Guaraní), and translation-based baselines whose MT system version was never recorded. Log per-language preprocessing explicitly — a single global "we lowercase and tokenize" line hides exactly the steps that differ across the languages you claim to cover.
A reproducibility manifest worth shipping
# repro-manifest.yml — include in the supplement
models:
- id: example-lm-7b, revision: a1b2c3d, dtype: bf16
- id: hosted-model-x, api_dates: 2026-05-02..2026-05-19
decoding: {temperature: 0.0, max_tokens: 512, samples: 1}
prompts: prompts/ # verbatim, one file per task x language
data:
- name: task_es, split_files: splits/es/, license: CC-BY-4.0
- name: task_gn, split_files: splits/gn/, license: community-terms
scoring: eval/score.py (chrF++ via sacrebleu 2.4.x, signature logged)
hardware: 4x A100-80GB, ~310 GPU-hours total
seeds: [13, 42, 2026] # every table reports mean/sd over these
known_gaps: human eval not re-runnable; transcripts included
The known_gaps line is the point: an honest boundary between re-runnable
and merely documented is what distinguishes a defensible checklist from a
hopeful one.
Where the investment pays: the response window
Reproducibility rigor is usually sold as ethics; at NAACL it is also tactics. When a reviewer asks "would the result hold with a different prompt phrasing?" a team with a pinned manifest and an experiment ledger answers inside the window with numbers; a team without one answers with adjectives. Concretely, the manifest converts three recurring review moments:
- A contamination worry becomes a lookup ("test items postdate the pinned revision's training cutoff") instead of a speculation.
- A "results seem fragile" score becomes challengeable with the seed spread already computed per table.
- A camera-ready promise becomes credible because the reviewer can see the infrastructure that would fulfill it.
Meta-reviews reward the second answer pattern visibly; the checklist is read as a proxy for whether the authors could defend any number under pressure.
Release levels, stated plainly
- Re-runnable — scripts + outputs + splits; a stranger reproduces the tables from the archive.
- Verifiable — outputs and scoring included; generation requires resources or access the reader may lack.
- Documented — full protocol description; execution not possible (private data, community-restricted corpora, retired APIs).
Name the level in the paper. NAACL reviewers penalize mismatch between claimed and actual level far more than they penalize level 3 honestly held — especially when community data-governance terms, common in Americas-language work, are the stated reason.
Output format
[Checklist audit] <answer -> evidence location -> holds/contradicted>
[Pin table status] <each moving part -> pinned/missing>
[Per-language gaps] <language -> unlogged preprocessing or scorer>
[Release level] re-runnable / verifiable / documented (+ reason)
[Fixes before upload] <ordered>