NeurIPS Reproducibility
Use this skill when a NeurIPS paper's claim depends on experiments, data, code, or a reproducibility
argument. The immediate target is a trustworthy main-track paper; the alternative route is MLRC/TMLR
when the central contribution is reproduction, replication, or generalizability of prior claims.
Main-track reproducibility bar
- State exact data splits, preprocessing, hyperparameters, selection criteria, compute resources,
software versions, and random-seed protocol.
- Report uncertainty where it matters: confidence intervals, standard errors, multiple seeds,
sensitivity checks, or negative findings.
- Distinguish exploratory experiments from evidence that supports the main claim.
- Make code/data availability match the checklist answer; "no" is allowed with justification, but a
central open-source benchmark or dataset usually needs accessible artifacts.
- For human, private, medical, proprietary, or safety-sensitive data, document access constraints
and ethical controls rather than pretending full release is possible.
MLRC route check
Consider the NeurIPS Reproducibility / MLRC track when the paper is primarily about confirming,
partially reproducing, failing to reproduce, or extending a published ML result. The 2026 MLRC route
requires TMLR review/acceptance before NeurIPS presentation consideration; this is not a shortcut
for ordinary main-track submissions.
Checklist-to-evidence cross-check
A "yes" on the NeurIPS Paper Checklist with nothing in the paper to back it is exactly what reviewers
hunt for. Run this cross-check so each reproducibility answer is honest and locatable; hedge the exact
item wording to the current year's checklist.
| Checklist answer |
Evidence that must exist |
Failure pattern reviewers flag |
| Code released: yes |
anonymous link plus run commands during review |
"yes" with no commands or a dead link |
| Data released: yes |
accessible split, license, and loading code |
central benchmark claimed open but not provided |
| Seeds/protocol reported |
seed count and aggregation rule in the text |
a single run reported as if deterministic |
| Compute reported |
hardware, wall-clock, and total resource budget |
omitted cost behind a "trained until converged" |
| Error bars reported |
intervals or std over runs on headline metrics |
bold-best numbers with no variance |
A justified "no" beats an unsupported "yes". If full release is blocked by privacy, licensing, or
safety, say so and document what reviewers can still verify.
Reviewer-pushback patterns
| Reviewer concern |
NeurIPS-specific fix |
| "Results may be a lucky seed" |
report multiple seeds with variance, not a single point |
| "Cannot rerun your pipeline" |
ship exact env, configs, and a one-command entry point in the ZIP |
| "Compute claims are unfair" |
disclose budget and tune baselines under the same budget |
| "Dataset access unclear" |
give license, hosting, and access steps, anonymized for review |
Worked vignette: a scaling-law claim
A paper claims a clean scaling law but reports one training run per model size with no intervals.
Reviewers cannot tell signal from seed noise. The fix before submission: add at least a few seeds at
the smaller sizes, plot variance bands, disclose the GPU-hours budget, and set the code-released and
error-bars checklist answers to a "yes" that the appendix actually supports. If the contribution were
instead reproducing someone else's published scaling law, the MLRC/TMLR route, not the main track,
would be the correct home.
Output format
[Reproducibility status] Strong / adequate / weak
[Claim at risk] <result that cannot yet be reproduced>
[Needed evidence] <code/data/seed/compute/ablation/error bars/license>
[Checklist changes] <items to revise>
[Route] Main track / MLRC-TMLR / other
Source: brycewang-stanford/Awesome-Journal-Skills → NeurIPS-Skills/skills/neurips-reproducibility/SKILL.md
1---2name: neurips-reproducibility3description: Use when strengthening NeurIPS reproducibility evidence, aligning Paper Checklist answers with the paper, writing code/data instructions, setting random-seed and compute disclosure, or deciding whether the MLRC/TMLR reproducibility route fits better than the main track or Datasets & Benchmarks track.4---567# NeurIPS Reproducibility89Use this skill when a NeurIPS paper's claim depends on experiments, data, code, or a reproducibility10argument. The immediate target is a trustworthy main-track paper; the alternative route is MLRC/TMLR11when the central contribution is reproduction, replication, or generalizability of prior claims.1213## Main-track reproducibility bar1415- State exact data splits, preprocessing, hyperparameters, selection criteria, compute resources,16 software versions, and random-seed protocol.17- Report uncertainty where it matters: confidence intervals, standard errors, multiple seeds,18 sensitivity checks, or negative findings.19- Distinguish exploratory experiments from evidence that supports the main claim.20- Make code/data availability match the checklist answer; "no" is allowed with justification, but a21 central open-source benchmark or dataset usually needs accessible artifacts.22- For human, private, medical, proprietary, or safety-sensitive data, document access constraints23 and ethical controls rather than pretending full release is possible.2425## MLRC route check2627Consider the NeurIPS Reproducibility / MLRC track when the paper is primarily about confirming,28partially reproducing, failing to reproduce, or extending a published ML result. The 2026 MLRC route29requires TMLR review/acceptance before NeurIPS presentation consideration; this is not a shortcut30for ordinary main-track submissions.3132## Checklist-to-evidence cross-check3334A "yes" on the NeurIPS Paper Checklist with nothing in the paper to back it is exactly what reviewers35hunt for. Run this cross-check so each reproducibility answer is honest and locatable; hedge the exact36item wording to the current year's checklist.3738| Checklist answer | Evidence that must exist | Failure pattern reviewers flag |39| --- | --- | --- |40| Code released: yes | anonymous link plus run commands during review | "yes" with no commands or a dead link |41| Data released: yes | accessible split, license, and loading code | central benchmark claimed open but not provided |42| Seeds/protocol reported | seed count and aggregation rule in the text | a single run reported as if deterministic |43| Compute reported | hardware, wall-clock, and total resource budget | omitted cost behind a "trained until converged" |44| Error bars reported | intervals or std over runs on headline metrics | bold-best numbers with no variance |4546A justified "no" beats an unsupported "yes". If full release is blocked by privacy, licensing, or47safety, say so and document what reviewers can still verify.4849## Reviewer-pushback patterns5051| Reviewer concern | NeurIPS-specific fix |52| --- | --- |53| "Results may be a lucky seed" | report multiple seeds with variance, not a single point |54| "Cannot rerun your pipeline" | ship exact env, configs, and a one-command entry point in the ZIP |55| "Compute claims are unfair" | disclose budget and tune baselines under the same budget |56| "Dataset access unclear" | give license, hosting, and access steps, anonymized for review |5758## Worked vignette: a scaling-law claim5960A paper claims a clean scaling law but reports one training run per model size with no intervals.61Reviewers cannot tell signal from seed noise. The fix before submission: add at least a few seeds at62the smaller sizes, plot variance bands, disclose the GPU-hours budget, and set the code-released and63error-bars checklist answers to a "yes" that the appendix actually supports. If the contribution were64instead reproducing someone else's published scaling law, the MLRC/TMLR route, not the main track,65would be the correct home.6667## Output format6869```text70[Reproducibility status] Strong / adequate / weak71[Claim at risk] <result that cannot yet be reproduced>72[Needed evidence] <code/data/seed/compute/ablation/error bars/license>73[Checklist changes] <items to revise>74[Route] Main track / MLRC-TMLR / other75```7677---7879**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `NeurIPS-Skills/skills/neurips-reproducibility/SKILL.md`
thedixitjain