AAAI Artifact Evaluation
Use this to prepare artifacts that reviewers can use to assess reproducibility. AAAI supplementary
material is part of the submission record; after review starts, do not assume it can be updated.
Artifact package
- Provide a technical appendix for proofs, algorithms, assumptions, hyperparameters, and extended
experiments.
- Provide code/data ZIPs that reproduce main tables or figures, with a short README, environment,
commands, seeds, expected outputs, and runtime.
- Provide multimedia appendices only when they support the technical claim.
- Remove author names, usernames, paths, repository history, cloud buckets, API keys, and metadata.
- Avoid web pointers in the reviewed submission unless current rules explicitly allow them.
- Include licensing and access notes for datasets, models, and third-party code.
AAAI-specific discipline
- Treat the supplementary deadline as final.
- Verify ZIP integrity before submission; missing or corrupted files may not be fixable during
rebuttal.
- Make the reproducibility checklist consistent with the artifact package.
- Prepare a post-acceptance public release path but keep review artifacts anonymous.
What an AAAI reviewer actually opens
AAAI does not run a separate badged artifact-evaluation committee the way some systems venues do;
the same broad-AI reviewer who scores the paper also inspects whatever supplement you attach. That
reviewer may be a planning, knowledge-representation, or constraint-satisfaction specialist rather
than a deep-learning engineer, so the artifact has to be legible without insider tooling. Optimize
for a reviewer who skims, not one who will spend an afternoon configuring a cluster.
| Reviewer action |
Passes |
Fails |
| Opens the ZIP |
sane tree, top README |
nested archives, 0-byte files |
| Reads appendix |
maps to numbered claims |
contradicts the paper |
| Tries one command |
reproduces one headline number |
needs private data or credentials |
| Scans for identity |
nothing reveals authors |
Git logs or home paths leak |
Phase-1 artifact red flags
Because clearly-below-bar papers can be cut before author feedback, a supplement that looks thin or
unrunnable is a cheap reason to summary-reject. Avoid these:
- Checklist promises released code, but the ZIP only holds figures and no scripts.
- A "see our repository" pointer to a mutable, deanonymizing URL.
- Multimedia attached for spectacle that carries no technical claim, inflating size with no rigor.
- Datasets shipped with no license note, leaving reuse legality unverifiable.
Worked vignette
A constraint-solving paper claims a 30% node-expansion reduction. The team ships a large ZIP of raw
solver logs but no driver script. The reproduction path is empty, so artifact status is "risky"; the
fix is a small run_main.py that regenerates Table 2 from seeds, a trimmed log sample, and a license
for the benchmark instances. The raw dump moves to the post-acceptance release.
Output format
[Artifact status] complete / partial / risky / unavailable
[Submitted files] technical appendix / multimedia appendix / code-data ZIP
[Reviewer reproduction path] <commands and expected output>
[Anonymity risks] <metadata, links, paths, logs>
[Missing items] <data, code, seeds, licenses, hardware>
1---2name: aaai-artifact-evaluation3description: Use when packaging AAAI code, data, multimedia appendices, technical appendices, reproducibility evidence, and post-acceptance artifact releases without violating double-blind or immutable-supplement rules.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/aaai-artifact-evaluation6---78# AAAI Artifact Evaluation910Use this to prepare artifacts that reviewers can use to assess reproducibility. AAAI supplementary11material is part of the submission record; after review starts, do not assume it can be updated.1213## Artifact package1415- Provide a technical appendix for proofs, algorithms, assumptions, hyperparameters, and extended16 experiments.17- Provide code/data ZIPs that reproduce main tables or figures, with a short README, environment,18 commands, seeds, expected outputs, and runtime.19- Provide multimedia appendices only when they support the technical claim.20- Remove author names, usernames, paths, repository history, cloud buckets, API keys, and metadata.21- Avoid web pointers in the reviewed submission unless current rules explicitly allow them.22- Include licensing and access notes for datasets, models, and third-party code.2324## AAAI-specific discipline2526- Treat the supplementary deadline as final.27- Verify ZIP integrity before submission; missing or corrupted files may not be fixable during28 rebuttal.29- Make the reproducibility checklist consistent with the artifact package.30- Prepare a post-acceptance public release path but keep review artifacts anonymous.3132## What an AAAI reviewer actually opens3334AAAI does not run a separate badged artifact-evaluation committee the way some systems venues do;35the same broad-AI reviewer who scores the paper also inspects whatever supplement you attach. That36reviewer may be a planning, knowledge-representation, or constraint-satisfaction specialist rather37than a deep-learning engineer, so the artifact has to be legible without insider tooling. Optimize38for a reviewer who skims, not one who will spend an afternoon configuring a cluster.3940| Reviewer action | Passes | Fails |41| --- | --- | --- |42| Opens the ZIP | sane tree, top README | nested archives, 0-byte files |43| Reads appendix | maps to numbered claims | contradicts the paper |44| Tries one command | reproduces one headline number | needs private data or credentials |45| Scans for identity | nothing reveals authors | Git logs or home paths leak |4647## Phase-1 artifact red flags4849Because clearly-below-bar papers can be cut before author feedback, a supplement that looks thin or50unrunnable is a cheap reason to summary-reject. Avoid these:5152- Checklist promises released code, but the ZIP only holds figures and no scripts.53- A "see our repository" pointer to a mutable, deanonymizing URL.54- Multimedia attached for spectacle that carries no technical claim, inflating size with no rigor.55- Datasets shipped with no license note, leaving reuse legality unverifiable.5657## Worked vignette5859A constraint-solving paper claims a 30% node-expansion reduction. The team ships a large ZIP of raw60solver logs but no driver script. The reproduction path is empty, so artifact status is "risky"; the61fix is a small `run_main.py` that regenerates Table 2 from seeds, a trimmed log sample, and a license62for the benchmark instances. The raw dump moves to the post-acceptance release.6364## Output format6566```text67[Artifact status] complete / partial / risky / unavailable68[Submitted files] technical appendix / multimedia appendix / code-data ZIP69[Reviewer reproduction path] <commands and expected output>70[Anonymity risks] <metadata, links, paths, logs>71[Missing items] <data, code, seeds, licenses, hardware>72```73