AISTATS Artifact Evaluation
Use this for evidence packaging around AISTATS. The venue centers on artificial intelligence,
statistics, and machine learning, so artifacts should make statistical and computational
claims inspectable.
Artifact plan
- Decide what evidence reviewers need: proof details, derivations, simulation scripts,
benchmark code, datasets, preprocessing, hyperparameter sweeps, random seeds, logs, or
qualitative examples.
- Keep decision-critical evidence in the main paper or appendix; optional run files can live
in supplementary material.
- Anonymize repository history, paths, notebook metadata, license headers, organization
names, cluster paths, grants, and commit authors.
- Include a minimal reproduction map: environment, dependencies, hardware, commands, expected
outputs, runtime, seeds, and known nondeterminism.
- For restricted data, give enough provenance and processing detail for credible
reproduction without violating data-use terms.
- After acceptance, replace anonymous archives with public, licensed, citable artifacts when
feasible.
What AISTATS evidence reviewers open first
| Claim type |
First artifact inspected |
Common failure caught |
| Convergence rate or regret bound |
Proof appendix and constants |
Condition used in the proof but missing from the theorem statement |
| Monte Carlo simulation |
Seeded simulation script |
Plots cannot be regenerated because seeds and replication counts are absent |
| Benchmark comparison |
Training and evaluation configs |
Baseline tuning budget undocumented |
| Bayesian or MCMC method |
Sampler diagnostics and chain logs |
No convergence statistics or trace evidence anywhere |
Because AISTATS reviewers are often statisticians, they will rerun a small simulation far
more readily than they will retrain a deep model, so make synthetic studies turnkey before
polishing anything else.
Worked vignette: packaging a Monte Carlo study
A hypothetical submission proposes a doubly robust treatment-effect estimator with a root-n
normality guarantee, validated on synthetic causal data plus two real benchmarks.
- Ship the data-generating process as one parameterized script rather than constants buried
in notebooks, so reviewers can vary n, dimension, and confounding strength.
- Record the replication count and the exact seed sequence used for every coverage and bias
table; AISTATS-style claims about interval coverage are meaningless without them.
- Emit tables directly from logged results so the PDF numbers and artifact numbers cannot
drift apart.
- State explicitly where the simulated regime satisfies the theorem assumptions and where it
deliberately violates them, since that mapping is what statistical reviewers grade.
Calibration anchors
- Supplementary inspection at AISTATS is at reviewer discretion; assume only the README and
one entry script get opened, and design accordingly.
- Upload size limits and accepted formats vary by cycle; verify against the current
OpenReview submission form rather than past years.
Output format
[Artifact role] anonymous supplement / camera-ready release / public archive
[Contents] <code/data/proofs/logs/notebooks>
[Anonymity risks] <paths/metadata/licenses/URLs>
[Reproduction level] turnkey / scripted / descriptive / weak
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → AISTATS-Skills/skills/aistats-artifact-evaluation/SKILL.md
1---2name: aistats-artifact-evaluation3description: Use when packaging AISTATS code, data, proofs, simulation scripts, notebooks, random seeds, and logs as anonymous supplementary evidence or public post-acceptance artifacts, even when there is no separate artifact badge. Covers what statistically minded AISTATS reviewers inspect first and how to make Monte Carlo studies turnkey.4---567# AISTATS Artifact Evaluation89Use this for evidence packaging around AISTATS. The venue centers on artificial intelligence,10statistics, and machine learning, so artifacts should make statistical and computational11claims inspectable.1213## Artifact plan1415- Decide what evidence reviewers need: proof details, derivations, simulation scripts,16 benchmark code, datasets, preprocessing, hyperparameter sweeps, random seeds, logs, or17 qualitative examples.18- Keep decision-critical evidence in the main paper or appendix; optional run files can live19 in supplementary material.20- Anonymize repository history, paths, notebook metadata, license headers, organization21 names, cluster paths, grants, and commit authors.22- Include a minimal reproduction map: environment, dependencies, hardware, commands, expected23 outputs, runtime, seeds, and known nondeterminism.24- For restricted data, give enough provenance and processing detail for credible25 reproduction without violating data-use terms.26- After acceptance, replace anonymous archives with public, licensed, citable artifacts when27 feasible.2829## What AISTATS evidence reviewers open first3031| Claim type | First artifact inspected | Common failure caught |32|---|---|---|33| Convergence rate or regret bound | Proof appendix and constants | Condition used in the proof but missing from the theorem statement |34| Monte Carlo simulation | Seeded simulation script | Plots cannot be regenerated because seeds and replication counts are absent |35| Benchmark comparison | Training and evaluation configs | Baseline tuning budget undocumented |36| Bayesian or MCMC method | Sampler diagnostics and chain logs | No convergence statistics or trace evidence anywhere |3738Because AISTATS reviewers are often statisticians, they will rerun a small simulation far39more readily than they will retrain a deep model, so make synthetic studies turnkey before40polishing anything else.4142## Worked vignette: packaging a Monte Carlo study4344A hypothetical submission proposes a doubly robust treatment-effect estimator with a root-n45normality guarantee, validated on synthetic causal data plus two real benchmarks.4647- Ship the data-generating process as one parameterized script rather than constants buried48 in notebooks, so reviewers can vary n, dimension, and confounding strength.49- Record the replication count and the exact seed sequence used for every coverage and bias50 table; AISTATS-style claims about interval coverage are meaningless without them.51- Emit tables directly from logged results so the PDF numbers and artifact numbers cannot52 drift apart.53- State explicitly where the simulated regime satisfies the theorem assumptions and where it54 deliberately violates them, since that mapping is what statistical reviewers grade.5556## Calibration anchors5758- Supplementary inspection at AISTATS is at reviewer discretion; assume only the README and59 one entry script get opened, and design accordingly.60- Upload size limits and accepted formats vary by cycle; verify against the current61 OpenReview submission form rather than past years.6263## Output format6465```text66[Artifact role] anonymous supplement / camera-ready release / public archive67[Contents] <code/data/proofs/logs/notebooks>68[Anonymity risks] <paths/metadata/licenses/URLs>69[Reproduction level] turnkey / scripted / descriptive / weak70[Fixes before upload] <ordered list>71```7273---7475**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `AISTATS-Skills/skills/aistats-artifact-evaluation/SKILL.md`
thedixitjain