NeurIPS Experiments
Use this skill before submission or rebuttal when the main question is whether the evidence supports
the NeurIPS claim. It is not enough to win a leaderboard; reviewers need to know why the result is
scientifically meaningful.
Experiment audit
- Baselines: include strong, current, tuned baselines and explain any missing comparison.
- Ablations: isolate the mechanism, not just remove components at random.
- Robustness: test across seeds, datasets, distribution shifts, scales, hyperparameters, or
realistic deployment conditions when relevant.
- Compute: disclose hardware, training time, resource assumptions, and whether comparisons are fair.
- Data: document splits, contamination controls, license, demographic or domain coverage, and
privacy/consent limits.
- Negative results: use them to calibrate claims; NeurIPS has a contribution type for negative
results, but the bar remains high.
- Use-inspired work: connect results to the real task without turning the paper into an application
report with no ML contribution.
Claim-to-evidence ladder
NeurIPS reviewers read experiments against the claim type. Put every headline claim on the ladder
before deciding whether the evidence is strong enough.
| Claim type |
Minimum evidence |
Strong evidence |
| Beats prior methods |
tuned current baselines, same data splits, comparable compute |
multiple suites, budget-matched tuning, significance or variance reporting |
| Mechanism explains gains |
ablation tied to the proposed mechanism |
intervention or diagnostic that rules out the obvious rival mechanism |
| Scales better |
at least two meaningful scales and fixed protocol |
trend across scales with compute, memory, and failure modes disclosed |
| Robust to distribution shift |
one out-of-domain or stress split |
several shifts with error analysis and narrowed claims where it fails |
| Useful in the real world |
task-relevant metric and realistic constraint |
deployment-like evaluation, safety/fairness/privacy caveats, and cost analysis |
| Negative result |
faithful implementation and fair reproduction attempt |
explains when the prior claim holds, fails, or needs qualification |
If the evidence only clears the minimum column, write the claim in minimum-column language. Reserve
strong-column language for results that survive the stronger checks.
Baseline fairness table
Before submission, make a table the reviewer could audit without trusting your narrative.
| Baseline issue |
Required disclosure |
Reviewer failure mode |
| Tuning budget |
search space, number of trials, early stopping, compute cap |
new method gets more tuning than baselines |
| Implementation source |
official code, reimplementation, or third-party fork |
weak reproduced baseline blamed on prior work |
| Data protocol |
splits, leakage checks, preprocessing, augmentation |
accidental train/test contamination |
| Resource match |
hardware, batch size, wall-clock, memory, total cost |
speed/accuracy tradeoff hidden |
| Selection rule |
validation metric and checkpoint choice |
cherry-picked best seed or test-set tuning |
Missing baselines are acceptable only when the omission is named and justified: unavailable code,
incompatible task, prohibitive compute, licensing, safety, or a scope mismatch. Do not silently omit
the strongest comparison.
Review-dimension stress test
Map the experimental section to the dimensions reviewers will naturally score.
| Dimension |
Experiment question |
| Quality |
Does the design isolate the proposed contribution rather than a confound? |
| Clarity |
Can a reader reproduce the protocol from the paper, appendix, and checklist? |
| Significance |
Does the effect matter beyond a narrow benchmark increment? |
| Contribution fit |
Do the experiments match General, Theory, Use-Inspired, Concept & Feasibility, or Negative Results framing? |
| Ethics/reproducibility |
Are data, compute, privacy, bias, and artifact limits disclosed honestly? |
Rebuttal triage gate
Not every missing experiment is rebuttal-feasible. Sort reviewer requests by value and risk.
| Request |
Rebuttal action |
Camera-ready or future-work action |
| Missing variance / seed concern |
run a small seed sweep or report existing variance |
expand seed grid if accepted |
| Missing obvious baseline |
add it only if implementation and tuning are fair in time |
otherwise explain omission and add after review |
| Mechanism unclear |
add a diagnostic ablation or error slice already supported by the code |
rewrite mechanism framing if evidence stays indirect |
| Dataset contamination worry |
add leakage check and describe split construction |
archive scripts and checklist support |
| New benchmark family |
usually too large for response unless already prepared |
narrow claim and schedule full evaluation later |
Rebuttal-ready evidence
Prepare small, high-signal clarifications that can fit in an author response: a missing baseline
table, a sanity check, an error analysis, a variance estimate, or a concise proof sketch. Do not
depend on a complete post-review paper rewrite.
Output format
[Evidence status] strong / adequate / weak
[Main unsupported claim] <claim>
[Critical missing experiment] <baseline/ablation/robustness/data/compute>
[Review dimension at risk] quality / clarity / significance / contribution fit / ethics-reproducibility
[Baseline fairness] tuned / comparable compute / same splits / missing justified
[Small rebuttal result] <result feasible during response>
[Claim rewrite] <narrower claim if evidence stays as is>
Source: brycewang-stanford/Awesome-Journal-Skills → NeurIPS-Skills/skills/neurips-experiments/SKILL.md
1---2name: neurips-experiments3description: Use when stress-testing NeurIPS experimental evidence, including baselines, ablations, data splits, compute, negative results, real-world use, and claim-to-evidence calibration.4---567# NeurIPS Experiments89Use this skill before submission or rebuttal when the main question is whether the evidence supports10the NeurIPS claim. It is not enough to win a leaderboard; reviewers need to know why the result is11scientifically meaningful.1213## Experiment audit1415- Baselines: include strong, current, tuned baselines and explain any missing comparison.16- Ablations: isolate the mechanism, not just remove components at random.17- Robustness: test across seeds, datasets, distribution shifts, scales, hyperparameters, or18 realistic deployment conditions when relevant.19- Compute: disclose hardware, training time, resource assumptions, and whether comparisons are fair.20- Data: document splits, contamination controls, license, demographic or domain coverage, and21 privacy/consent limits.22- Negative results: use them to calibrate claims; NeurIPS has a contribution type for negative23 results, but the bar remains high.24- Use-inspired work: connect results to the real task without turning the paper into an application25 report with no ML contribution.2627## Claim-to-evidence ladder2829NeurIPS reviewers read experiments against the claim type. Put every headline claim on the ladder30before deciding whether the evidence is strong enough.3132| Claim type | Minimum evidence | Strong evidence |33| --- | --- | --- |34| Beats prior methods | tuned current baselines, same data splits, comparable compute | multiple suites, budget-matched tuning, significance or variance reporting |35| Mechanism explains gains | ablation tied to the proposed mechanism | intervention or diagnostic that rules out the obvious rival mechanism |36| Scales better | at least two meaningful scales and fixed protocol | trend across scales with compute, memory, and failure modes disclosed |37| Robust to distribution shift | one out-of-domain or stress split | several shifts with error analysis and narrowed claims where it fails |38| Useful in the real world | task-relevant metric and realistic constraint | deployment-like evaluation, safety/fairness/privacy caveats, and cost analysis |39| Negative result | faithful implementation and fair reproduction attempt | explains when the prior claim holds, fails, or needs qualification |4041If the evidence only clears the minimum column, write the claim in minimum-column language. Reserve42strong-column language for results that survive the stronger checks.4344## Baseline fairness table4546Before submission, make a table the reviewer could audit without trusting your narrative.4748| Baseline issue | Required disclosure | Reviewer failure mode |49| --- | --- | --- |50| Tuning budget | search space, number of trials, early stopping, compute cap | new method gets more tuning than baselines |51| Implementation source | official code, reimplementation, or third-party fork | weak reproduced baseline blamed on prior work |52| Data protocol | splits, leakage checks, preprocessing, augmentation | accidental train/test contamination |53| Resource match | hardware, batch size, wall-clock, memory, total cost | speed/accuracy tradeoff hidden |54| Selection rule | validation metric and checkpoint choice | cherry-picked best seed or test-set tuning |5556Missing baselines are acceptable only when the omission is named and justified: unavailable code,57incompatible task, prohibitive compute, licensing, safety, or a scope mismatch. Do not silently omit58the strongest comparison.5960## Review-dimension stress test6162Map the experimental section to the dimensions reviewers will naturally score.6364| Dimension | Experiment question |65| --- | --- |66| Quality | Does the design isolate the proposed contribution rather than a confound? |67| Clarity | Can a reader reproduce the protocol from the paper, appendix, and checklist? |68| Significance | Does the effect matter beyond a narrow benchmark increment? |69| Contribution fit | Do the experiments match General, Theory, Use-Inspired, Concept & Feasibility, or Negative Results framing? |70| Ethics/reproducibility | Are data, compute, privacy, bias, and artifact limits disclosed honestly? |7172## Rebuttal triage gate7374Not every missing experiment is rebuttal-feasible. Sort reviewer requests by value and risk.7576| Request | Rebuttal action | Camera-ready or future-work action |77| --- | --- | --- |78| Missing variance / seed concern | run a small seed sweep or report existing variance | expand seed grid if accepted |79| Missing obvious baseline | add it only if implementation and tuning are fair in time | otherwise explain omission and add after review |80| Mechanism unclear | add a diagnostic ablation or error slice already supported by the code | rewrite mechanism framing if evidence stays indirect |81| Dataset contamination worry | add leakage check and describe split construction | archive scripts and checklist support |82| New benchmark family | usually too large for response unless already prepared | narrow claim and schedule full evaluation later |8384## Rebuttal-ready evidence8586Prepare small, high-signal clarifications that can fit in an author response: a missing baseline87table, a sanity check, an error analysis, a variance estimate, or a concise proof sketch. Do not88depend on a complete post-review paper rewrite.8990## Output format9192```text93[Evidence status] strong / adequate / weak94[Main unsupported claim] <claim>95[Critical missing experiment] <baseline/ablation/robustness/data/compute>96[Review dimension at risk] quality / clarity / significance / contribution fit / ethics-reproducibility97[Baseline fairness] tuned / comparable compute / same splits / missing justified98[Small rebuttal result] <result feasible during response>99[Claim rewrite] <narrower claim if evidence stays as is>100```101102---103104**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `NeurIPS-Skills/skills/neurips-experiments/SKILL.md`
thedixitjain