AAAI Reproducibility
Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-27 requires the
reproducibility checklist to be uploaded separately from the main PDF, in its own field on the
submission form (AAAI-26 carried it inside the PDF after the references) — so it is a document a
reviewer opens on its own, and it has to agree with the paper and supplement rather than read as an
afterthought. AAAI-27 also states that reviewers assess reproducibility from what was actually
submitted, and that material promised "after acceptance or publication" is not evidence it exists.
Reproducibility audit
- Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item,
checklist answer, or code/data artifact.
- Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt
selection, and hardware.
- Report variance or uncertainty when stochasticity affects conclusions.
- Document dataset licenses, access constraints, sensitive data, human-subjects issues, and
annotation procedures.
- Separate training compute, inference compute, and experiment search cost.
- Check the reproducibility checklist for contradictions with the main text and supplement.
Common AAAI weaknesses
- Checklist says code/data are available but supplement lacks runnable commands.
- Main results rely on one seed, one benchmark, or one prompt family.
- Baselines are weaker than current open-source or widely cited systems.
- Evaluation uses closed data or APIs with no reproducibility substitute.
- Human evaluation omits annotator instructions or quality control.
Checklist-to-evidence consistency grid
AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes"
against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no",
because it signals the checklist was filled in carelessly.
| Checklist answer |
Must be backed by |
Phase-1 risk if unbacked |
| code available |
runnable scripts in the ZIP |
"claimed but absent" |
| seeds reported |
seed list and variance |
"single-run cherry-pick" |
| compute disclosed |
train vs. inference vs. search cost |
"hidden tuning budget" |
| data accessible |
license and access path |
"irreproducible by anyone" |
Claim-evidence ledger
Create a row for every claim that appears in the abstract, introduction, or conclusion. The ledger
should be short enough to audit before submission and concrete enough that a Phase-1 reviewer can see
that each headline claim is checkable.
| Ledger field |
What to record |
Common failure |
| Claim text |
exact sentence or paraphrase from the paper |
claim becomes stronger than the evidence |
| Evidence artifact |
theorem, table, figure, appendix, code command, data sheet, or log path |
evidence exists but is not submitted |
| Reproducibility inputs |
seeds, splits, prompts, preprocessing, hardware, hyperparameters, and model versions |
rerun cannot recreate the result |
| Variance and controls |
confidence interval, standard deviation, multiple seeds, ablation, or matched-compute baseline |
single lucky run drives the claim |
| Checklist answer |
the checklist item whose answer depends on this artifact |
checklist contradicts the supplement |
| Reviewer risk |
what a skeptical reviewer would challenge first |
rebuttal cannot fix missing evidence |
For each row, choose one of three actions: keep the claim because the artifact is present,
weaken the claim to match the evidence, or add the missing artifact before submission. Do not
leave a row in "promise later" state.
Artifact dry-run
Before upload, run the artifact as if the reviewer has no private context:
- Unzip the submitted package into a clean directory.
- Read only the included README, not local lab notes.
- Run the smallest command that regenerates one headline table or figure.
- Check that expected runtime, hardware, random seeds, data download/access, and license constraints
are stated before the command.
- Confirm that output files have deterministic names and map back to paper tables.
- Mark any non-runnable or restricted component as such in both the README and checklist.
The dry-run can be small; it does not need to reproduce every experiment. Its purpose is to prove that
the submitted artifact is not merely decorative and that the checklist answers are honest.
Reviewer-pushback patterns
- "Checklist says code available but I see only figures." Fix: ship scripts and a one-line driver
before the deadline; do not promise the repository in rebuttal.
- "Results may be seed-dependent." Fix: report multiple seeds with spread, and set the checklist seed
answer to match the supplement exactly.
- "Closed API, not reproducible." Fix: add an open substitute model or release prompts and outputs so
the claim is checkable.
Worked vignette
A vision-language paper checks "code and data available" but the ZIP holds only PDFs of plots. Audit
verdict: reproducibility grade "fragile", with a checklist conflict between the "yes" and the missing
scripts. The smallest fix is a reproduce.sh that regenerates one headline table from seeds plus a
dataset license note, after which the checklist answer becomes truthful and Phase-1 defensible.
Output format
[Reproducibility grade] strong / adequate / fragile / not reviewable
[Checklist conflicts] <answers that contradict paper/supplement>
[Evidence gaps] <claims without submitted verification>
[Compute/data disclosure] complete / incomplete
[Priority fixes] <smallest changes before submission>
1---2name: aaai-reproducibility3description: Use when strengthening an AAAI paper's reproducibility checklist (placed after references), experimental traceability, seed and hyperparameter reporting, compute and cost disclosure, dataset access and licensing, code/data ZIP readiness, and the claim-to-evidence map that Phase-1 reviewers use to judge rigor across AAAI's broad AI scope.4---56# AAAI Reproducibility78Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-27 requires the9reproducibility checklist to be **uploaded separately** from the main PDF, in its own field on the10submission form (AAAI-26 carried it inside the PDF after the references) — so it is a document a11reviewer opens on its own, and it has to agree with the paper and supplement rather than read as an12afterthought. AAAI-27 also states that reviewers assess reproducibility from what was actually13submitted, and that material promised "after acceptance or publication" is not evidence it exists.1415## Reproducibility audit1617- Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item,18 checklist answer, or code/data artifact.19- Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt20 selection, and hardware.21- Report variance or uncertainty when stochasticity affects conclusions.22- Document dataset licenses, access constraints, sensitive data, human-subjects issues, and23 annotation procedures.24- Separate training compute, inference compute, and experiment search cost.25- Check the reproducibility checklist for contradictions with the main text and supplement.2627## Common AAAI weaknesses2829- Checklist says code/data are available but supplement lacks runnable commands.30- Main results rely on one seed, one benchmark, or one prompt family.31- Baselines are weaker than current open-source or widely cited systems.32- Evaluation uses closed data or APIs with no reproducibility substitute.33- Human evaluation omits annotator instructions or quality control.3435## Checklist-to-evidence consistency grid3637AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes"38against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no",39because it signals the checklist was filled in carelessly.4041| Checklist answer | Must be backed by | Phase-1 risk if unbacked |42| --- | --- | --- |43| code available | runnable scripts in the ZIP | "claimed but absent" |44| seeds reported | seed list and variance | "single-run cherry-pick" |45| compute disclosed | train vs. inference vs. search cost | "hidden tuning budget" |46| data accessible | license and access path | "irreproducible by anyone" |4748## Claim-evidence ledger4950Create a row for every claim that appears in the abstract, introduction, or conclusion. The ledger51should be short enough to audit before submission and concrete enough that a Phase-1 reviewer can see52that each headline claim is checkable.5354| Ledger field | What to record | Common failure |55| --- | --- | --- |56| Claim text | exact sentence or paraphrase from the paper | claim becomes stronger than the evidence |57| Evidence artifact | theorem, table, figure, appendix, code command, data sheet, or log path | evidence exists but is not submitted |58| Reproducibility inputs | seeds, splits, prompts, preprocessing, hardware, hyperparameters, and model versions | rerun cannot recreate the result |59| Variance and controls | confidence interval, standard deviation, multiple seeds, ablation, or matched-compute baseline | single lucky run drives the claim |60| Checklist answer | the checklist item whose answer depends on this artifact | checklist contradicts the supplement |61| Reviewer risk | what a skeptical reviewer would challenge first | rebuttal cannot fix missing evidence |6263For each row, choose one of three actions: **keep** the claim because the artifact is present,64**weaken** the claim to match the evidence, or **add** the missing artifact before submission. Do not65leave a row in "promise later" state.6667## Artifact dry-run6869Before upload, run the artifact as if the reviewer has no private context:70711. Unzip the submitted package into a clean directory.722. Read only the included README, not local lab notes.733. Run the smallest command that regenerates one headline table or figure.744. Check that expected runtime, hardware, random seeds, data download/access, and license constraints75 are stated before the command.765. Confirm that output files have deterministic names and map back to paper tables.776. Mark any non-runnable or restricted component as such in both the README and checklist.7879The dry-run can be small; it does not need to reproduce every experiment. Its purpose is to prove that80the submitted artifact is not merely decorative and that the checklist answers are honest.8182## Reviewer-pushback patterns8384- "Checklist says code available but I see only figures." Fix: ship scripts and a one-line driver85 before the deadline; do not promise the repository in rebuttal.86- "Results may be seed-dependent." Fix: report multiple seeds with spread, and set the checklist seed87 answer to match the supplement exactly.88- "Closed API, not reproducible." Fix: add an open substitute model or release prompts and outputs so89 the claim is checkable.9091## Worked vignette9293A vision-language paper checks "code and data available" but the ZIP holds only PDFs of plots. Audit94verdict: reproducibility grade "fragile", with a checklist conflict between the "yes" and the missing95scripts. The smallest fix is a `reproduce.sh` that regenerates one headline table from seeds plus a96dataset license note, after which the checklist answer becomes truthful and Phase-1 defensible.9798## Output format99100```text101[Reproducibility grade] strong / adequate / fragile / not reviewable102[Checklist conflicts] <answers that contradict paper/supplement>103[Evidence gaps] <claims without submitted verification>104[Compute/data disclosure] complete / incomplete105[Priority fixes] <smallest changes before submission>106```
brycewang-stanford