NeurIPS Experiments
Use this skill before submission or rebuttal when the main question is whether the evidence supports
the NeurIPS claim. It is not enough to win a leaderboard; reviewers need to know why the result is
scientifically meaningful.
Experiment audit
- Baselines: include strong, current, tuned baselines and explain any missing comparison.
- Ablations: isolate the mechanism, not just remove components at random.
- Robustness: test across seeds, datasets, distribution shifts, scales, hyperparameters, or
realistic deployment conditions when relevant.
- Compute: disclose hardware, training time, resource assumptions, and whether comparisons are fair.
- Data: document splits, contamination controls, license, demographic or domain coverage, and
privacy/consent limits.
- Negative results: use them to calibrate claims; NeurIPS has a contribution type for negative
results, but the bar remains high.
- Use-inspired work: connect results to the real task without turning the paper into an application
report with no ML contribution.
Claim-to-evidence ladder
NeurIPS reviewers read experiments against the claim type. Put every headline claim on the ladder
before deciding whether the evidence is strong enough.
| Claim type |
Minimum evidence |
Strong evidence |
| Beats prior methods |
tuned current baselines, same data splits, comparable compute |
multiple suites, budget-matched tuning, significance or variance reporting |
| Mechanism explains gains |
ablation tied to the proposed mechanism |
intervention or diagnostic that rules out the obvious rival mechanism |
| Scales better |
at least two meaningful scales and fixed protocol |
trend across scales with compute, memory, and failure modes disclosed |
| Robust to distribution shift |
one out-of-domain or stress split |
several shifts with error analysis and narrowed claims where it fails |
| Useful in the real world |
task-relevant metric and realistic constraint |
deployment-like evaluation, safety/fairness/privacy caveats, and cost analysis |
| Negative result |
faithful implementation and fair reproduction attempt |
explains when the prior claim holds, fails, or needs qualification |
If the evidence only clears the minimum column, write the claim in minimum-column language. Reserve
strong-column language for results that survive the stronger checks.
Baseline fairness table
Before submission, make a table the reviewer could audit without trusting your narrative.
| Baseline issue |
Required disclosure |
Reviewer failure mode |
| Tuning budget |
search space, number of trials, early stopping, compute cap |
new method gets more tuning than baselines |
| Implementation source |
official code, reimplementation, or third-party fork |
weak reproduced baseline blamed on prior work |
| Data protocol |
splits, leakage checks, preprocessing, augmentation |
accidental train/test contamination |
| Resource match |
hardware, batch size, wall-clock, memory, total cost |
speed/accuracy tradeoff hidden |
| Selection rule |
validation metric and checkpoint choice |
cherry-picked best seed or test-set tuning |
Missing baselines are acceptable only when the omission is named and justified: unavailable code,
incompatible task, prohibitive compute, licensing, safety, or a scope mismatch. Do not silently omit
the strongest comparison.
Review-dimension stress test
Map the experimental section to the dimensions reviewers will naturally score.
| Dimension |
Experiment question |
| Quality |
Does the design isolate the proposed contribution rather than a confound? |
| Clarity |
Can a reader reproduce the protocol from the paper, appendix, and checklist? |
| Significance |
Does the effect matter beyond a narrow benchmark increment? |
| Contribution fit |
Do the experiments match General, Theory, Use-Inspired, Concept & Feasibility, or Negative Results framing? |
| Ethics/reproducibility |
Are data, compute, privacy, bias, and artifact limits disclosed honestly? |
Rebuttal triage gate
Not every missing experiment is rebuttal-feasible. Sort reviewer requests by value and risk.
| Request |
Rebuttal action |
Camera-ready or future-work action |
| Missing variance / seed concern |
run a small seed sweep or report existing variance |
expand seed grid if accepted |
| Missing obvious baseline |
add it only if implementation and tuning are fair in time |
otherwise explain omission and add after review |
| Mechanism unclear |
add a diagnostic ablation or error slice already supported by the code |
rewrite mechanism framing if evidence stays indirect |
| Dataset contamination worry |
add leakage check and describe split construction |
archive scripts and checklist support |
| New benchmark family |
usually too large for response unless already prepared |
narrow claim and schedule full evaluation later |
Rebuttal-ready evidence
Prepare small, high-signal clarifications that can fit in an author response: a missing baseline
table, a sanity check, an error analysis, a variance estimate, or a concise proof sketch. Do not
depend on a complete post-review paper rewrite.
Output format
[Evidence status] strong / adequate / weak
[Main unsupported claim] <claim>
[Critical missing experiment] <baseline/ablation/robustness/data/compute>
[Review dimension at risk] quality / clarity / significance / contribution fit / ethics-reproducibility
[Baseline fairness] tuned / comparable compute / same splits / missing justified
[Small rebuttal result] <result feasible during response>
[Claim rewrite] <narrower claim if evidence stays as is>
1---2name: neurips-experiments3description: Use when stress-testing NeurIPS experimental evidence, including baselines, ablations, data splits, compute, negative results, real-world use, and claim-to-evidence calibration.4---56# NeurIPS Experiments78Use this skill before submission or rebuttal when the main question is whether the evidence supports9the NeurIPS claim. It is not enough to win a leaderboard; reviewers need to know why the result is10scientifically meaningful.1112## Experiment audit1314- Baselines: include strong, current, tuned baselines and explain any missing comparison.15- Ablations: isolate the mechanism, not just remove components at random.16- Robustness: test across seeds, datasets, distribution shifts, scales, hyperparameters, or17 realistic deployment conditions when relevant.18- Compute: disclose hardware, training time, resource assumptions, and whether comparisons are fair.19- Data: document splits, contamination controls, license, demographic or domain coverage, and20 privacy/consent limits.21- Negative results: use them to calibrate claims; NeurIPS has a contribution type for negative22 results, but the bar remains high.23- Use-inspired work: connect results to the real task without turning the paper into an application24 report with no ML contribution.2526## Claim-to-evidence ladder2728NeurIPS reviewers read experiments against the claim type. Put every headline claim on the ladder29before deciding whether the evidence is strong enough.3031| Claim type | Minimum evidence | Strong evidence |32| --- | --- | --- |33| Beats prior methods | tuned current baselines, same data splits, comparable compute | multiple suites, budget-matched tuning, significance or variance reporting |34| Mechanism explains gains | ablation tied to the proposed mechanism | intervention or diagnostic that rules out the obvious rival mechanism |35| Scales better | at least two meaningful scales and fixed protocol | trend across scales with compute, memory, and failure modes disclosed |36| Robust to distribution shift | one out-of-domain or stress split | several shifts with error analysis and narrowed claims where it fails |37| Useful in the real world | task-relevant metric and realistic constraint | deployment-like evaluation, safety/fairness/privacy caveats, and cost analysis |38| Negative result | faithful implementation and fair reproduction attempt | explains when the prior claim holds, fails, or needs qualification |3940If the evidence only clears the minimum column, write the claim in minimum-column language. Reserve41strong-column language for results that survive the stronger checks.4243## Baseline fairness table4445Before submission, make a table the reviewer could audit without trusting your narrative.4647| Baseline issue | Required disclosure | Reviewer failure mode |48| --- | --- | --- |49| Tuning budget | search space, number of trials, early stopping, compute cap | new method gets more tuning than baselines |50| Implementation source | official code, reimplementation, or third-party fork | weak reproduced baseline blamed on prior work |51| Data protocol | splits, leakage checks, preprocessing, augmentation | accidental train/test contamination |52| Resource match | hardware, batch size, wall-clock, memory, total cost | speed/accuracy tradeoff hidden |53| Selection rule | validation metric and checkpoint choice | cherry-picked best seed or test-set tuning |5455Missing baselines are acceptable only when the omission is named and justified: unavailable code,56incompatible task, prohibitive compute, licensing, safety, or a scope mismatch. Do not silently omit57the strongest comparison.5859## Review-dimension stress test6061Map the experimental section to the dimensions reviewers will naturally score.6263| Dimension | Experiment question |64| --- | --- |65| Quality | Does the design isolate the proposed contribution rather than a confound? |66| Clarity | Can a reader reproduce the protocol from the paper, appendix, and checklist? |67| Significance | Does the effect matter beyond a narrow benchmark increment? |68| Contribution fit | Do the experiments match General, Theory, Use-Inspired, Concept & Feasibility, or Negative Results framing? |69| Ethics/reproducibility | Are data, compute, privacy, bias, and artifact limits disclosed honestly? |7071## Rebuttal triage gate7273Not every missing experiment is rebuttal-feasible. Sort reviewer requests by value and risk.7475| Request | Rebuttal action | Camera-ready or future-work action |76| --- | --- | --- |77| Missing variance / seed concern | run a small seed sweep or report existing variance | expand seed grid if accepted |78| Missing obvious baseline | add it only if implementation and tuning are fair in time | otherwise explain omission and add after review |79| Mechanism unclear | add a diagnostic ablation or error slice already supported by the code | rewrite mechanism framing if evidence stays indirect |80| Dataset contamination worry | add leakage check and describe split construction | archive scripts and checklist support |81| New benchmark family | usually too large for response unless already prepared | narrow claim and schedule full evaluation later |8283## Rebuttal-ready evidence8485Prepare small, high-signal clarifications that can fit in an author response: a missing baseline86table, a sanity check, an error analysis, a variance estimate, or a concise proof sketch. Do not87depend on a complete post-review paper rewrite.8889## Output format9091```text92[Evidence status] strong / adequate / weak93[Main unsupported claim] <claim>94[Critical missing experiment] <baseline/ablation/robustness/data/compute>95[Review dimension at risk] quality / clarity / significance / contribution fit / ethics-reproducibility96[Baseline fairness] tuned / comparable compute / same splits / missing justified97[Small rebuttal result] <result feasible during response>98[Claim rewrite] <narrower claim if evidence stays as is>99```
brycewang-stanford