AAAI Reproducibility
Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-26 required a
reproducibility checklist after references, so the checklist must agree with the paper and
supplement rather than read as an afterthought.
Reproducibility audit
- Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item,
checklist answer, or code/data artifact.
- Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt
selection, and hardware.
- Report variance or uncertainty when stochasticity affects conclusions.
- Document dataset licenses, access constraints, sensitive data, human-subjects issues, and
annotation procedures.
- Separate training compute, inference compute, and experiment search cost.
- Check the reproducibility checklist for contradictions with the main text and supplement.
Common AAAI weaknesses
- Checklist says code/data are available but supplement lacks runnable commands.
- Main results rely on one seed, one benchmark, or one prompt family.
- Baselines are weaker than current open-source or widely cited systems.
- Evaluation uses closed data or APIs with no reproducibility substitute.
- Human evaluation omits annotator instructions or quality control.
Checklist-to-evidence consistency grid
AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes"
against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no",
because it signals the checklist was filled in carelessly.
| Checklist answer |
Must be backed by |
Phase-1 risk if unbacked |
| code available |
runnable scripts in the ZIP |
"claimed but absent" |
| seeds reported |
seed list and variance |
"single-run cherry-pick" |
| compute disclosed |
train vs. inference vs. search cost |
"hidden tuning budget" |
| data accessible |
license and access path |
"irreproducible by anyone" |
Claim-evidence ledger
Create a row for every claim that appears in the abstract, introduction, or conclusion. The ledger
should be short enough to audit before submission and concrete enough that a Phase-1 reviewer can see
that each headline claim is checkable.
| Ledger field |
What to record |
Common failure |
| Claim text |
exact sentence or paraphrase from the paper |
claim becomes stronger than the evidence |
| Evidence artifact |
theorem, table, figure, appendix, code command, data sheet, or log path |
evidence exists but is not submitted |
| Reproducibility inputs |
seeds, splits, prompts, preprocessing, hardware, hyperparameters, and model versions |
rerun cannot recreate the result |
| Variance and controls |
confidence interval, standard deviation, multiple seeds, ablation, or matched-compute baseline |
single lucky run drives the claim |
| Checklist answer |
the checklist item whose answer depends on this artifact |
checklist contradicts the supplement |
| Reviewer risk |
what a skeptical reviewer would challenge first |
rebuttal cannot fix missing evidence |
For each row, choose one of three actions: keep the claim because the artifact is present,
weaken the claim to match the evidence, or add the missing artifact before submission. Do not
leave a row in "promise later" state.
Artifact dry-run
Before upload, run the artifact as if the reviewer has no private context:
- Unzip the submitted package into a clean directory.
- Read only the included README, not local lab notes.
- Run the smallest command that regenerates one headline table or figure.
- Check that expected runtime, hardware, random seeds, data download/access, and license constraints
are stated before the command.
- Confirm that output files have deterministic names and map back to paper tables.
- Mark any non-runnable or restricted component as such in both the README and checklist.
The dry-run can be small; it does not need to reproduce every experiment. Its purpose is to prove that
the submitted artifact is not merely decorative and that the checklist answers are honest.
Reviewer-pushback patterns
- "Checklist says code available but I see only figures." Fix: ship scripts and a one-line driver
before the deadline; do not promise the repository in rebuttal.
- "Results may be seed-dependent." Fix: report multiple seeds with spread, and set the checklist seed
answer to match the supplement exactly.
- "Closed API, not reproducible." Fix: add an open substitute model or release prompts and outputs so
the claim is checkable.
Worked vignette
A vision-language paper checks "code and data available" but the ZIP holds only PDFs of plots. Audit
verdict: reproducibility grade "fragile", with a checklist conflict between the "yes" and the missing
scripts. The smallest fix is a reproduce.sh that regenerates one headline table from seeds plus a
dataset license note, after which the checklist answer becomes truthful and Phase-1 defensible.
Output format
[Reproducibility grade] strong / adequate / fragile / not reviewable
[Checklist conflicts] <answers that contradict paper/supplement>
[Evidence gaps] <claims without submitted verification>
[Compute/data disclosure] complete / incomplete
[Priority fixes] <smallest changes before submission>
Source: brycewang-stanford/Awesome-Journal-Skills → AAAI-Skills/skills/aaai-reproducibility/SKILL.md
1---2name: aaai-reproducibility3description: Use when strengthening an AAAI paper's reproducibility checklist (placed after references), experimental traceability, seed and hyperparameter reporting, compute and cost disclosure, dataset access and licensing, code/data ZIP readiness, and the claim-to-evidence map that Phase-1 reviewers use to judge rigor across AAAI's broad AI scope.4---567# AAAI Reproducibility89Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-26 required a10reproducibility checklist after references, so the checklist must agree with the paper and11supplement rather than read as an afterthought.1213## Reproducibility audit1415- Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item,16 checklist answer, or code/data artifact.17- Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt18 selection, and hardware.19- Report variance or uncertainty when stochasticity affects conclusions.20- Document dataset licenses, access constraints, sensitive data, human-subjects issues, and21 annotation procedures.22- Separate training compute, inference compute, and experiment search cost.23- Check the reproducibility checklist for contradictions with the main text and supplement.2425## Common AAAI weaknesses2627- Checklist says code/data are available but supplement lacks runnable commands.28- Main results rely on one seed, one benchmark, or one prompt family.29- Baselines are weaker than current open-source or widely cited systems.30- Evaluation uses closed data or APIs with no reproducibility substitute.31- Human evaluation omits annotator instructions or quality control.3233## Checklist-to-evidence consistency grid3435AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes"36against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no",37because it signals the checklist was filled in carelessly.3839| Checklist answer | Must be backed by | Phase-1 risk if unbacked |40| --- | --- | --- |41| code available | runnable scripts in the ZIP | "claimed but absent" |42| seeds reported | seed list and variance | "single-run cherry-pick" |43| compute disclosed | train vs. inference vs. search cost | "hidden tuning budget" |44| data accessible | license and access path | "irreproducible by anyone" |4546## Claim-evidence ledger4748Create a row for every claim that appears in the abstract, introduction, or conclusion. The ledger49should be short enough to audit before submission and concrete enough that a Phase-1 reviewer can see50that each headline claim is checkable.5152| Ledger field | What to record | Common failure |53| --- | --- | --- |54| Claim text | exact sentence or paraphrase from the paper | claim becomes stronger than the evidence |55| Evidence artifact | theorem, table, figure, appendix, code command, data sheet, or log path | evidence exists but is not submitted |56| Reproducibility inputs | seeds, splits, prompts, preprocessing, hardware, hyperparameters, and model versions | rerun cannot recreate the result |57| Variance and controls | confidence interval, standard deviation, multiple seeds, ablation, or matched-compute baseline | single lucky run drives the claim |58| Checklist answer | the checklist item whose answer depends on this artifact | checklist contradicts the supplement |59| Reviewer risk | what a skeptical reviewer would challenge first | rebuttal cannot fix missing evidence |6061For each row, choose one of three actions: **keep** the claim because the artifact is present,62**weaken** the claim to match the evidence, or **add** the missing artifact before submission. Do not63leave a row in "promise later" state.6465## Artifact dry-run6667Before upload, run the artifact as if the reviewer has no private context:68691. Unzip the submitted package into a clean directory.702. Read only the included README, not local lab notes.713. Run the smallest command that regenerates one headline table or figure.724. Check that expected runtime, hardware, random seeds, data download/access, and license constraints73 are stated before the command.745. Confirm that output files have deterministic names and map back to paper tables.756. Mark any non-runnable or restricted component as such in both the README and checklist.7677The dry-run can be small; it does not need to reproduce every experiment. Its purpose is to prove that78the submitted artifact is not merely decorative and that the checklist answers are honest.7980## Reviewer-pushback patterns8182- "Checklist says code available but I see only figures." Fix: ship scripts and a one-line driver83 before the deadline; do not promise the repository in rebuttal.84- "Results may be seed-dependent." Fix: report multiple seeds with spread, and set the checklist seed85 answer to match the supplement exactly.86- "Closed API, not reproducible." Fix: add an open substitute model or release prompts and outputs so87 the claim is checkable.8889## Worked vignette9091A vision-language paper checks "code and data available" but the ZIP holds only PDFs of plots. Audit92verdict: reproducibility grade "fragile", with a checklist conflict between the "yes" and the missing93scripts. The smallest fix is a `reproduce.sh` that regenerates one headline table from seeds plus a94dataset license note, after which the checklist answer becomes truthful and Phase-1 defensible.9596## Output format9798```text99[Reproducibility grade] strong / adequate / fragile / not reviewable100[Checklist conflicts] <answers that contradict paper/supplement>101[Evidence gaps] <claims without submitted verification>102[Compute/data disclosure] complete / incomplete103[Priority fixes] <smallest changes before submission>104```105106---107108**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `AAAI-Skills/skills/aaai-reproducibility/SKILL.md`
thedixitjain