ICLR Experiments
Use this before submission or during a revision pass to stress-test empirical claims. ICLR
experiments should answer the scientific question, not merely assemble a leaderboard.
Experiment audit
- Match each experiment to a claim in the introduction.
- Compare against current strong baselines, open-source systems, and the most relevant recent
OpenReview/arXiv papers.
- Add ablations that isolate one mechanism at a time.
- Report variance across seeds or runs when randomness can change conclusions.
- Include robustness checks for dataset shift, prompt changes, architecture variants, hyperparameter
sensitivity, or compute scale when those affect the claim.
- State compute budget, hardware, training time, inference cost, and environmental or access limits
where relevant.
- For human evaluation, document task, annotator instructions, aggregation, quality control, and IRB
or ethics status when needed.
Reviewer questions to pre-answer
- Is the baseline tuned fairly?
- Does the method win because of more compute, data, parameters, or prompt search?
- Does the effect persist outside the easiest benchmark?
- Are negative results hidden?
- Can a reviewer reproduce the headline table from the supplement or artifact?
What ICLR reviewers reward in evidence
ICLR's empirical culture prizes honest ablations and mechanism over leaderboard position. A clean
ablation that explains why a representation works often outscores a larger raw number.
| Claim type |
Evidence that convinces ICLR reviewers |
Common reject trigger |
| New objective helps |
Ablate the objective with everything else fixed |
Gains confounded with extra tuning |
| Method scales |
Several model sizes/tasks with a trend |
One large run, no scaling curve |
| Robust representation |
Tests across shifts, seeds, prompts |
Single-seed peak on one benchmark |
| Beats prior method |
Tuned, current, open-source baseline |
Stale or under-tuned baseline |
Worked vignette
A paper claims a new self-supervised pretext task yields better linear-probe accuracy. Reviewers
ask whether the gain is the pretext task or simply longer pretraining. The author audit: hold total
pretraining compute fixed, swap only the pretext objective, and report linear-probe accuracy with
error bars over five seeds. The compute-matched ablation isolates the mechanism and is small enough
to post inline during discussion, where the table becomes part of the permanent public record.
Reviewer-pushback patterns
- "You win because of more compute." Add a compute-matched control; report FLOPs, not just wall time.
- "Only one seed." Report mean and spread across seeds; an unstable benchmark needs variance.
- "Baseline is weak." Cite the baseline's own recommended settings and show you matched them.
- "Ablation removes two things at once." Split into single-mechanism ablations a reviewer can read.
Output format
[Claim] <paper claim>
[Experiment evidence] sufficient / needs baseline / needs ablation / needs robustness
[Fairness issue] <compute, tuning, data, prompt, metric>
[Fast fix] <experiment or analysis feasible before deadline>
[Appendix placement] <what can move out of main text>
Source: brycewang-stanford/Awesome-Journal-Skills → ICLR-Skills/skills/iclr-experiments/SKILL.md
1---2name: iclr-experiments3description: Use when designing or auditing ICLR experiments, including baselines, ablations, scaling laws, robustness, statistics, benchmarks, human evaluation, and compute reporting. Use when a reviewer questions whether a representation-learning or model gain is real, when you must isolate one mechanism with an ablation, or when preparing a small compute-matched control that can be posted inline during the public discussion period.4---567# ICLR Experiments89Use this before submission or during a revision pass to stress-test empirical claims. ICLR10experiments should answer the scientific question, not merely assemble a leaderboard.1112## Experiment audit1314- Match each experiment to a claim in the introduction.15- Compare against current strong baselines, open-source systems, and the most relevant recent16 OpenReview/arXiv papers.17- Add ablations that isolate one mechanism at a time.18- Report variance across seeds or runs when randomness can change conclusions.19- Include robustness checks for dataset shift, prompt changes, architecture variants, hyperparameter20 sensitivity, or compute scale when those affect the claim.21- State compute budget, hardware, training time, inference cost, and environmental or access limits22 where relevant.23- For human evaluation, document task, annotator instructions, aggregation, quality control, and IRB24 or ethics status when needed.2526## Reviewer questions to pre-answer2728- Is the baseline tuned fairly?29- Does the method win because of more compute, data, parameters, or prompt search?30- Does the effect persist outside the easiest benchmark?31- Are negative results hidden?32- Can a reviewer reproduce the headline table from the supplement or artifact?3334## What ICLR reviewers reward in evidence3536ICLR's empirical culture prizes honest ablations and mechanism over leaderboard position. A clean37ablation that explains *why* a representation works often outscores a larger raw number.3839| Claim type | Evidence that convinces ICLR reviewers | Common reject trigger |40| --- | --- | --- |41| New objective helps | Ablate the objective with everything else fixed | Gains confounded with extra tuning |42| Method scales | Several model sizes/tasks with a trend | One large run, no scaling curve |43| Robust representation | Tests across shifts, seeds, prompts | Single-seed peak on one benchmark |44| Beats prior method | Tuned, current, open-source baseline | Stale or under-tuned baseline |4546## Worked vignette4748A paper claims a new self-supervised pretext task yields better linear-probe accuracy. Reviewers49ask whether the gain is the pretext task or simply longer pretraining. The author audit: hold total50pretraining compute fixed, swap only the pretext objective, and report linear-probe accuracy with51error bars over five seeds. The compute-matched ablation isolates the mechanism and is small enough52to post inline during discussion, where the table becomes part of the permanent public record.5354## Reviewer-pushback patterns5556- "You win because of more compute." Add a compute-matched control; report FLOPs, not just wall time.57- "Only one seed." Report mean and spread across seeds; an unstable benchmark needs variance.58- "Baseline is weak." Cite the baseline's own recommended settings and show you matched them.59- "Ablation removes two things at once." Split into single-mechanism ablations a reviewer can read.6061## Output format6263```text64[Claim] <paper claim>65[Experiment evidence] sufficient / needs baseline / needs ablation / needs robustness66[Fairness issue] <compute, tuning, data, prompt, metric>67[Fast fix] <experiment or analysis feasible before deadline>68[Appendix placement] <what can move out of main text>69```7071---7273**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `ICLR-Skills/skills/iclr-experiments/SKILL.md`
thedixitjain