ICML Experiments
Use this before submission or rebuttal when the central issue is whether experiments are sound
enough for ICML. The question is not just "does it win"; it is whether the evidence supports the ML
claim under fair comparison.
Experiment audit
- Baselines: current, strong, tuned, and correctly implemented.
- Ablations: isolate mechanism, architecture, objective, data, or optimization change.
- Variance: report seeds, confidence intervals, standard deviations, or a reason variance is not
meaningful.
- Data: check leakage, split construction, duplication, filtering, licensing, and representative
coverage.
- Compute: disclose hardware, training cost, inference cost, and comparison fairness.
- Scaling: show whether gains persist across model sizes, datasets, horizons, or domains when that
supports the claim.
- Negative results: use failures to define boundaries rather than hide them.
- Appendix: put supporting detail there, but keep decisive evidence in the main 8 pages.
Reviewer-pushback patterns and the ICML fix
| Pushback |
Why it lands at ICML |
Fix |
| "Convergence guarantees under assumptions the experiments violate" |
Theory paper asserts a rate under smoothness or bounded variance, but the deep-learning runs break it |
State assumptions honestly, add a figure showing the rate holds empirically in-regime, flag where it does not |
| "Missing strong, tuned baselines" |
The leaderboard win used an undertuned competitor |
Re-tune the baseline with matched budget, report the search protocol |
| "No variance, single seed" |
One run cannot separate signal from noise |
Report seeds with confidence intervals or justify determinism |
| "Compute not disclosed" |
ICML expects hardware and training-cost transparency |
Add a compute table and confirm comparison fairness |
Worked vignette: optimizer claim audit
A paper claims a new adaptive step-size method beats Adam with a non-convex convergence guarantee.
The audit asks: is Adam tuned with the same budget, do the benchmark losses actually satisfy the
proof's assumptions, and do gains survive across seeds and model sizes? If the win shrinks under a
tuned baseline or the assumptions hold only on toy quadratics, the right move is to narrow the claim
to the regime where both theory and experiments agree, rather than overclaim a universal speedup.
Rebuttal-ready result
During response, prefer a small decisive table, corrected baseline, missing ablation, or concise
error analysis over a broad new experimental section. ICML gives one discussion round, so a single
tuned-baseline row or in-regime variance plot moves a reviewer more than a sprawling new study.
Output format
[Evidence status] strong / adequate / weak
[Most vulnerable claim] <claim>
[Critical missing result] <baseline/ablation/variance/leakage/compute>
[Small response result] <feasible clarification>
[Claim narrowing] <text if evidence is not enough>
Source: brycewang-stanford/Awesome-Journal-Skills → ICML-Skills/skills/icml-experiments/SKILL.md
1---2name: icml-experiments3description: Use when stress-testing ICML experimental evidence before submission or rebuttal, including strong tuned baselines, mechanism-isolating ablations, seed variance and confidence intervals, compute disclosure, data leakage and split construction, reproducibility, negative results, and fit to ICML soundness, originality, and significance scoring.4---5
6
7# ICML Experiments
8
9Use this before submission or rebuttal when the central issue is whether experiments are sound
10enough for ICML. The question is not just "does it win"; it is whether the evidence supports the ML
11claim under fair comparison.
12
13## Experiment audit
14
15- Baselines: current, strong, tuned, and correctly implemented.
16- Ablations: isolate mechanism, architecture, objective, data, or optimization change.
17- Variance: report seeds, confidence intervals, standard deviations, or a reason variance is not
18 meaningful.
19- Data: check leakage, split construction, duplication, filtering, licensing, and representative
20 coverage.
21- Compute: disclose hardware, training cost, inference cost, and comparison fairness.
22- Scaling: show whether gains persist across model sizes, datasets, horizons, or domains when that
23 supports the claim.
24- Negative results: use failures to define boundaries rather than hide them.
25- Appendix: put supporting detail there, but keep decisive evidence in the main 8 pages.
26
27## Reviewer-pushback patterns and the ICML fix
28
29| Pushback | Why it lands at ICML | Fix |
30| --- | --- | --- |
31| "Convergence guarantees under assumptions the experiments violate" | Theory paper asserts a rate under smoothness or bounded variance, but the deep-learning runs break it | State assumptions honestly, add a figure showing the rate holds empirically in-regime, flag where it does not |
32| "Missing strong, tuned baselines" | The leaderboard win used an undertuned competitor | Re-tune the baseline with matched budget, report the search protocol |
33| "No variance, single seed" | One run cannot separate signal from noise | Report seeds with confidence intervals or justify determinism |
34| "Compute not disclosed" | ICML expects hardware and training-cost transparency | Add a compute table and confirm comparison fairness |
35
36## Worked vignette: optimizer claim audit
37
38A paper claims a new adaptive step-size method beats Adam with a non-convex convergence guarantee.
39The audit asks: is Adam tuned with the same budget, do the benchmark losses actually satisfy the
40proof's assumptions, and do gains survive across seeds and model sizes? If the win shrinks under a
41tuned baseline or the assumptions hold only on toy quadratics, the right move is to narrow the claim
42to the regime where both theory and experiments agree, rather than overclaim a universal speedup.
43
44## Rebuttal-ready result
45
46During response, prefer a small decisive table, corrected baseline, missing ablation, or concise
47error analysis over a broad new experimental section. ICML gives one discussion round, so a single
48tuned-baseline row or in-regime variance plot moves a reviewer more than a sprawling new study.
49
50## Output format
51
52```text
53[Evidence status] strong / adequate / weak
54[Most vulnerable claim] <claim>
55[Critical missing result] <baseline/ablation/variance/leakage/compute>
56[Small response result] <feasible clarification>
57[Claim narrowing] <text if evidence is not enough>
58```
59
60---
61
62**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `ICML-Skills/skills/icml-experiments/SKILL.md`
thedixitjain