ICML Experiments
Use this before submission or rebuttal when the central issue is whether experiments are sound
enough for ICML. The question is not just "does it win"; it is whether the evidence supports the ML
claim under fair comparison.
Experiment audit
- Baselines: current, strong, tuned, and correctly implemented.
- Ablations: isolate mechanism, architecture, objective, data, or optimization change.
- Variance: report seeds, confidence intervals, standard deviations, or a reason variance is not
meaningful.
- Data: check leakage, split construction, duplication, filtering, licensing, and representative
coverage.
- Compute: disclose hardware, training cost, inference cost, and comparison fairness.
- Scaling: show whether gains persist across model sizes, datasets, horizons, or domains when that
supports the claim.
- Negative results: use failures to define boundaries rather than hide them.
- Appendix: put supporting detail there, but keep decisive evidence in the main 8 pages.
Reviewer-pushback patterns and the ICML fix
| Pushback |
Why it lands at ICML |
Fix |
| "Convergence guarantees under assumptions the experiments violate" |
Theory paper asserts a rate under smoothness or bounded variance, but the deep-learning runs break it |
State assumptions honestly, add a figure showing the rate holds empirically in-regime, flag where it does not |
| "Missing strong, tuned baselines" |
The leaderboard win used an undertuned competitor |
Re-tune the baseline with matched budget, report the search protocol |
| "No variance, single seed" |
One run cannot separate signal from noise |
Report seeds with confidence intervals or justify determinism |
| "Compute not disclosed" |
ICML expects hardware and training-cost transparency |
Add a compute table and confirm comparison fairness |
Worked vignette: optimizer claim audit
A paper claims a new adaptive step-size method beats Adam with a non-convex convergence guarantee.
The audit asks: is Adam tuned with the same budget, do the benchmark losses actually satisfy the
proof's assumptions, and do gains survive across seeds and model sizes? If the win shrinks under a
tuned baseline or the assumptions hold only on toy quadratics, the right move is to narrow the claim
to the regime where both theory and experiments agree, rather than overclaim a universal speedup.
Rebuttal-ready result
During response, prefer a small decisive table, corrected baseline, missing ablation, or concise
error analysis over a broad new experimental section. ICML gives one discussion round, so a single
tuned-baseline row or in-regime variance plot moves a reviewer more than a sprawling new study.
Output format
[Evidence status] strong / adequate / weak
[Most vulnerable claim] <claim>
[Critical missing result] <baseline/ablation/variance/leakage/compute>
[Small response result] <feasible clarification>
[Claim narrowing] <text if evidence is not enough>
1---2name: icml-experiments3description: Use when stress-testing ICML experimental evidence before submission or rebuttal, including strong tuned baselines, mechanism-isolating ablations, seed variance and confidence intervals, compute disclosure, data leakage and split construction, reproducibility, negative results, and fit to ICML soundness, originality, and significance scoring.4---56# ICML Experiments78Use this before submission or rebuttal when the central issue is whether experiments are sound9enough for ICML. The question is not just "does it win"; it is whether the evidence supports the ML10claim under fair comparison.1112## Experiment audit1314- Baselines: current, strong, tuned, and correctly implemented.15- Ablations: isolate mechanism, architecture, objective, data, or optimization change.16- Variance: report seeds, confidence intervals, standard deviations, or a reason variance is not17 meaningful.18- Data: check leakage, split construction, duplication, filtering, licensing, and representative19 coverage.20- Compute: disclose hardware, training cost, inference cost, and comparison fairness.21- Scaling: show whether gains persist across model sizes, datasets, horizons, or domains when that22 supports the claim.23- Negative results: use failures to define boundaries rather than hide them.24- Appendix: put supporting detail there, but keep decisive evidence in the main 8 pages.2526## Reviewer-pushback patterns and the ICML fix2728| Pushback | Why it lands at ICML | Fix |29| --- | --- | --- |30| "Convergence guarantees under assumptions the experiments violate" | Theory paper asserts a rate under smoothness or bounded variance, but the deep-learning runs break it | State assumptions honestly, add a figure showing the rate holds empirically in-regime, flag where it does not |31| "Missing strong, tuned baselines" | The leaderboard win used an undertuned competitor | Re-tune the baseline with matched budget, report the search protocol |32| "No variance, single seed" | One run cannot separate signal from noise | Report seeds with confidence intervals or justify determinism |33| "Compute not disclosed" | ICML expects hardware and training-cost transparency | Add a compute table and confirm comparison fairness |3435## Worked vignette: optimizer claim audit3637A paper claims a new adaptive step-size method beats Adam with a non-convex convergence guarantee.38The audit asks: is Adam tuned with the same budget, do the benchmark losses actually satisfy the39proof's assumptions, and do gains survive across seeds and model sizes? If the win shrinks under a40tuned baseline or the assumptions hold only on toy quadratics, the right move is to narrow the claim41to the regime where both theory and experiments agree, rather than overclaim a universal speedup.4243## Rebuttal-ready result4445During response, prefer a small decisive table, corrected baseline, missing ablation, or concise46error analysis over a broad new experimental section. ICML gives one discussion round, so a single47tuned-baseline row or in-regime variance plot moves a reviewer more than a sprawling new study.4849## Output format5051```text52[Evidence status] strong / adequate / weak53[Most vulnerable claim] <claim>54[Critical missing result] <baseline/ablation/variance/leakage/compute>55[Small response result] <feasible clarification>56[Claim narrowing] <text if evidence is not enough>57```58
brycewang-stanford