ICML Reproducibility
Use this when the paper's acceptance risk is tied to whether experiments, code, or theory can be
trusted. ICML reviewers are asked to evaluate soundness, and ICML author instructions state that
reproducibility and code availability are considered in decisions.
Evidence checklist
- Data: source, license, preprocessing, splits, leakage checks, and access restrictions.
- Code: anonymous review package, environment, dependencies, exact commands, and expected runtime.
- Randomness: seeds, variance, confidence intervals, or explanation for deterministic runs.
- Compute: hardware, training budget, evaluation budget, and fairness relative to baselines.
- Baselines: tuning protocol, implementation source, and why missing baselines are not applicable.
- Theory: assumptions, theorem statements, proof dependencies, and edge cases.
- Impact: support claims in the impact statement with real evidence or narrow the statement.
ICML-specific public-record issue
Accepted papers may publish original supplementary material. Do not put unreleasable data, identity
leaks, or private credentials in the review package. If data cannot be public, document the access
path and ethics constraints in the paper.
Reproducibility scoring lens
ICML reviewers fold reproducibility into the soundness judgment rather than scoring it on a separate
axis, so the question is whether a skeptical reviewer could regenerate the headline number.
| Signal a reviewer checks |
Strong evidence |
Weak signal that invites doubt |
| Code in review package |
Anonymized, runnable, exact commands |
"Code on acceptance" promise only |
| Variance |
Seeds with intervals |
Single run, no spread |
| Compute |
Hardware and budget table |
Unstated cost, unfair comparison |
| Theory |
Assumptions and proof dependencies listed |
Theorem with hidden conditions |
Worked vignette: theory-plus-benchmark paper
For an adaptive-optimizer paper, reproducibility means the convergence proof's assumptions are
written out, the benchmark scripts run from the anonymized supplement with a fixed seed, and the
compute table lets a reviewer judge whether the speedup is real or a tuning artifact. The recurring
failure is a clean theorem paired with benchmark code that silently relies on an unreleased internal
dataset; document the access path or move to a public dataset before the deadline.
Reviewer-pushback patterns and the ICML fix
| Pushback |
ICML-specific fix |
| "Cannot reproduce without the code" |
Ship the anonymized runnable package now, not a post-acceptance promise |
| "Hyperparameters undocumented" |
Add the search protocol and final values to the appendix |
| "Speedup may be a seed artifact" |
Report multiple seeds with confidence intervals |
| "Theorem assumptions hidden" |
List assumptions, proof dependencies, and edge cases explicitly |
Because accepted ICML papers can publish the original supplementary material on OpenReview, the
reproducibility package is also a public commitment. Confirm before the deadline that every file is
releasable, every license is stated, and no private credential or identity path remains.
Output format
[Reproducibility status] strong / adequate / weak
[Weakest claim] <claim not yet supported>
[Required fix] <code/data/seed/compute/baseline/proof>
[Supplement/public-record risk] <none or issue>
[Reviewer-facing sentence] <concise reproducibility statement>
1---2name: icml-reproducibility3description: Use when strengthening ICML reproducibility evidence, including code/data availability, random seeds, compute disclosure, appendix evidence, impact-statement support, and reviewer-facing reproducibility claims.4---56# ICML Reproducibility78Use this when the paper's acceptance risk is tied to whether experiments, code, or theory can be9trusted. ICML reviewers are asked to evaluate soundness, and ICML author instructions state that10reproducibility and code availability are considered in decisions.1112## Evidence checklist1314- Data: source, license, preprocessing, splits, leakage checks, and access restrictions.15- Code: anonymous review package, environment, dependencies, exact commands, and expected runtime.16- Randomness: seeds, variance, confidence intervals, or explanation for deterministic runs.17- Compute: hardware, training budget, evaluation budget, and fairness relative to baselines.18- Baselines: tuning protocol, implementation source, and why missing baselines are not applicable.19- Theory: assumptions, theorem statements, proof dependencies, and edge cases.20- Impact: support claims in the impact statement with real evidence or narrow the statement.2122## ICML-specific public-record issue2324Accepted papers may publish original supplementary material. Do not put unreleasable data, identity25leaks, or private credentials in the review package. If data cannot be public, document the access26path and ethics constraints in the paper.2728## Reproducibility scoring lens2930ICML reviewers fold reproducibility into the soundness judgment rather than scoring it on a separate31axis, so the question is whether a skeptical reviewer could regenerate the headline number.3233| Signal a reviewer checks | Strong evidence | Weak signal that invites doubt |34| --- | --- | --- |35| Code in review package | Anonymized, runnable, exact commands | "Code on acceptance" promise only |36| Variance | Seeds with intervals | Single run, no spread |37| Compute | Hardware and budget table | Unstated cost, unfair comparison |38| Theory | Assumptions and proof dependencies listed | Theorem with hidden conditions |3940## Worked vignette: theory-plus-benchmark paper4142For an adaptive-optimizer paper, reproducibility means the convergence proof's assumptions are43written out, the benchmark scripts run from the anonymized supplement with a fixed seed, and the44compute table lets a reviewer judge whether the speedup is real or a tuning artifact. The recurring45failure is a clean theorem paired with benchmark code that silently relies on an unreleased internal46dataset; document the access path or move to a public dataset before the deadline.4748## Reviewer-pushback patterns and the ICML fix4950| Pushback | ICML-specific fix |51| --- | --- |52| "Cannot reproduce without the code" | Ship the anonymized runnable package now, not a post-acceptance promise |53| "Hyperparameters undocumented" | Add the search protocol and final values to the appendix |54| "Speedup may be a seed artifact" | Report multiple seeds with confidence intervals |55| "Theorem assumptions hidden" | List assumptions, proof dependencies, and edge cases explicitly |5657Because accepted ICML papers can publish the original supplementary material on OpenReview, the58reproducibility package is also a public commitment. Confirm before the deadline that every file is59releasable, every license is stated, and no private credential or identity path remains.6061## Output format6263```text64[Reproducibility status] strong / adequate / weak65[Weakest claim] <claim not yet supported>66[Required fix] <code/data/seed/compute/baseline/proof>67[Supplement/public-record risk] <none or issue>68[Reviewer-facing sentence] <concise reproducibility statement>69```70
brycewang-stanford