NeurIPS Reproducibility
Use this skill when a NeurIPS paper's claim depends on experiments, data, code, or a reproducibility
argument. The immediate target is a trustworthy main-track paper; the alternative route is MLRC/TMLR
when the central contribution is reproduction, replication, or generalizability of prior claims.
Main-track reproducibility bar
- State exact data splits, preprocessing, hyperparameters, selection criteria, compute resources,
software versions, and random-seed protocol.
- Report uncertainty where it matters: confidence intervals, standard errors, multiple seeds,
sensitivity checks, or negative findings.
- Distinguish exploratory experiments from evidence that supports the main claim.
- Make code/data availability match the checklist answer; "no" is allowed with justification, but a
central open-source benchmark or dataset usually needs accessible artifacts.
- For human, private, medical, proprietary, or safety-sensitive data, document access constraints
and ethical controls rather than pretending full release is possible.
MLRC route check
Consider the NeurIPS Reproducibility / MLRC track when the paper is primarily about confirming,
partially reproducing, failing to reproduce, or extending a published ML result. The 2026 MLRC route
requires TMLR review/acceptance before NeurIPS presentation consideration; this is not a shortcut
for ordinary main-track submissions.
Checklist-to-evidence cross-check
A "yes" on the NeurIPS Paper Checklist with nothing in the paper to back it is exactly what reviewers
hunt for. Run this cross-check so each reproducibility answer is honest and locatable; hedge the exact
item wording to the current year's checklist.
| Checklist answer |
Evidence that must exist |
Failure pattern reviewers flag |
| Code released: yes |
anonymous link plus run commands during review |
"yes" with no commands or a dead link |
| Data released: yes |
accessible split, license, and loading code |
central benchmark claimed open but not provided |
| Seeds/protocol reported |
seed count and aggregation rule in the text |
a single run reported as if deterministic |
| Compute reported |
hardware, wall-clock, and total resource budget |
omitted cost behind a "trained until converged" |
| Error bars reported |
intervals or std over runs on headline metrics |
bold-best numbers with no variance |
A justified "no" beats an unsupported "yes". If full release is blocked by privacy, licensing, or
safety, say so and document what reviewers can still verify.
Reviewer-pushback patterns
| Reviewer concern |
NeurIPS-specific fix |
| "Results may be a lucky seed" |
report multiple seeds with variance, not a single point |
| "Cannot rerun your pipeline" |
ship exact env, configs, and a one-command entry point in the ZIP |
| "Compute claims are unfair" |
disclose budget and tune baselines under the same budget |
| "Dataset access unclear" |
give license, hosting, and access steps, anonymized for review |
Worked vignette: a scaling-law claim
A paper claims a clean scaling law but reports one training run per model size with no intervals.
Reviewers cannot tell signal from seed noise. The fix before submission: add at least a few seeds at
the smaller sizes, plot variance bands, disclose the GPU-hours budget, and set the code-released and
error-bars checklist answers to a "yes" that the appendix actually supports. If the contribution were
instead reproducing someone else's published scaling law, the MLRC/TMLR route, not the main track,
would be the correct home.
Output format
[Reproducibility status] Strong / adequate / weak
[Claim at risk] <result that cannot yet be reproduced>
[Needed evidence] <code/data/seed/compute/ablation/error bars/license>
[Checklist changes] <items to revise>
[Route] Main track / MLRC-TMLR / other
1---2name: neurips-reproducibility3description: Use when strengthening NeurIPS reproducibility evidence, aligning Paper Checklist answers with the paper, writing code/data instructions, setting random-seed and compute disclosure, or deciding whether the MLRC/TMLR reproducibility route fits better than the main track or Datasets & Benchmarks track.4---56# NeurIPS Reproducibility78Use this skill when a NeurIPS paper's claim depends on experiments, data, code, or a reproducibility9argument. The immediate target is a trustworthy main-track paper; the alternative route is MLRC/TMLR10when the central contribution is reproduction, replication, or generalizability of prior claims.1112## Main-track reproducibility bar1314- State exact data splits, preprocessing, hyperparameters, selection criteria, compute resources,15 software versions, and random-seed protocol.16- Report uncertainty where it matters: confidence intervals, standard errors, multiple seeds,17 sensitivity checks, or negative findings.18- Distinguish exploratory experiments from evidence that supports the main claim.19- Make code/data availability match the checklist answer; "no" is allowed with justification, but a20 central open-source benchmark or dataset usually needs accessible artifacts.21- For human, private, medical, proprietary, or safety-sensitive data, document access constraints22 and ethical controls rather than pretending full release is possible.2324## MLRC route check2526Consider the NeurIPS Reproducibility / MLRC track when the paper is primarily about confirming,27partially reproducing, failing to reproduce, or extending a published ML result. The 2026 MLRC route28requires TMLR review/acceptance before NeurIPS presentation consideration; this is not a shortcut29for ordinary main-track submissions.3031## Checklist-to-evidence cross-check3233A "yes" on the NeurIPS Paper Checklist with nothing in the paper to back it is exactly what reviewers34hunt for. Run this cross-check so each reproducibility answer is honest and locatable; hedge the exact35item wording to the current year's checklist.3637| Checklist answer | Evidence that must exist | Failure pattern reviewers flag |38| --- | --- | --- |39| Code released: yes | anonymous link plus run commands during review | "yes" with no commands or a dead link |40| Data released: yes | accessible split, license, and loading code | central benchmark claimed open but not provided |41| Seeds/protocol reported | seed count and aggregation rule in the text | a single run reported as if deterministic |42| Compute reported | hardware, wall-clock, and total resource budget | omitted cost behind a "trained until converged" |43| Error bars reported | intervals or std over runs on headline metrics | bold-best numbers with no variance |4445A justified "no" beats an unsupported "yes". If full release is blocked by privacy, licensing, or46safety, say so and document what reviewers can still verify.4748## Reviewer-pushback patterns4950| Reviewer concern | NeurIPS-specific fix |51| --- | --- |52| "Results may be a lucky seed" | report multiple seeds with variance, not a single point |53| "Cannot rerun your pipeline" | ship exact env, configs, and a one-command entry point in the ZIP |54| "Compute claims are unfair" | disclose budget and tune baselines under the same budget |55| "Dataset access unclear" | give license, hosting, and access steps, anonymized for review |5657## Worked vignette: a scaling-law claim5859A paper claims a clean scaling law but reports one training run per model size with no intervals.60Reviewers cannot tell signal from seed noise. The fix before submission: add at least a few seeds at61the smaller sizes, plot variance bands, disclose the GPU-hours budget, and set the code-released and62error-bars checklist answers to a "yes" that the appendix actually supports. If the contribution were63instead reproducing someone else's published scaling law, the MLRC/TMLR route, not the main track,64would be the correct home.6566## Output format6768```text69[Reproducibility status] Strong / adequate / weak70[Claim at risk] <result that cannot yet be reproduced>71[Needed evidence] <code/data/seed/compute/ablation/error bars/license>72[Checklist changes] <items to revise>73[Route] Main track / MLRC-TMLR / other74```75
brycewang-stanford