brycewang-stanford UAI Artifact Evaluation
UAI posted no separate artifact-evaluation track or badge system for the 2026 cycle (existence of one in later cycles: 待核实). What it did post is sharper than a badge: code and data release is strongly encouraged, a 50 MB anonymous ZIP rides with the submission, and reviewers judge whether claims are "backed up convincingly." The artifact's job is to convert a skeptical probabilistic-ML reviewer's spot-check into confirmation.
Design for the five-minute skeptic
Because reviewers are explicitly not required to consult supplementary material, assume whoever opens the ZIP gives it five minutes. Optimize for that reader:
- A top-level
READMEthat states, in its first ten lines, what claim each script reproduces and how long it takes. - One command per headline result, each with a small-scale mode that finishes on a laptop: a reviewer who reproduces Figure 2 at n=500 in ninety seconds will extend you trust for the n=50,000 version.
- Pinned dependencies and one canonical entry point. A
Makefileor single driver script beats a directory of loose notebooks. - Expected outputs stored beside the scripts, so "did it work?" needs no judgment.
What inference artifacts must expose
Probabilistic-modeling papers fail reproduction in venue-specific ways. Package accordingly:
| Contribution type | Must be in the artifact | Spot-check the reviewer will try |
|---|---|---|
| MCMC / SMC sampler | Seeds, chain configs, convergence diagnostics code | Re-run short chains; compare R-hat and ESS to reported |
| Variational method | ELBO logging, initialization scheme, optimizer settings | Reproduce the ELBO trace shape, not just the endpoint |
| Causal discovery | Graph-generation scripts, SHD/SID evaluation code | Regenerate synthetic graphs; verify metrics on one seed |
| Calibration / conformal | Score functions, split definitions, coverage computation | Recompute empirical coverage at one α |
| Probabilistic programming / PGM inference | Model source, query set, exact-baseline harness | Run the exact baseline on the smallest instance |
| Theory with simulations | Every constant used to instantiate the bound | Check simulation matches theorem conditions |
Anonymity inside the archive
The double-blind requirement covers all supplementary material explicitly. Archive leaks are quieter than PDF leaks, so build from a clean export:
# Build an anonymous artifact from a clean tree, never from the working repo
git archive --format=tar HEAD | tar -x -C /tmp/uai-artifact
cd /tmp/uai-artifact
grep -rniE 'university|\.edu|author|thanks|grant' --include='*.py' --include='*.md' . | head
find . -name '*.ipynb' -exec grep -l '"authors"' {} \; # notebook metadata
rm -rf .git .github; find . -name '.DS_Store' -delete
zip -r ../supplement.zip . && du -h ../supplement.zip # must be ≤ 50 MB (2026 cap)
Watch for: license headers with names, pyproject.toml author fields, conda environment
exports embedding usernames, wandb/MLflow run URLs tied to an account, dataset paths
containing /home/<name>/, and model checkpoints whose training config JSON names a
cluster.
README skeleton for the archive
The archive's README is the artifact's abstract. A shape that fits the five-minute reader:
# Supplementary code for submission #<OpenReview number>
## Claim → command map (small-scale modes; full-scale flags noted)
| Paper claim | Command | Runtime (laptop) |
|---|---|---|
| Fig. 2 (coverage vs n) | `make fig2-small` | ~2 min |
| Table 1 (SHD, 10 seeds) | `make table1-small` | ~4 min |
| Thm. 3 simulation | `make thm3-check` | ~1 min |
## Environment
- `pip install -r requirements.txt` (pinned versions; Python 3.11)
- No GPU required for small-scale modes.
## Expected outputs
- `expected/` holds reference CSVs; each command prints PASS/FAIL against them.
## Full-scale reproduction
- Flags, hardware used, and total runtime per experiment in `FULL_RUNS.md`.
Everything above stays anonymous by construction — no names, no lab conventions in paths, no acknowledgement of infrastructure that identifies an institution.
Data that cannot ship
- Under the 50 MB cap, ship generators, not corpora: the script that produces the synthetic SCMs or the subsampled benchmark, plus checksums for the full data.
- For licensed or private datasets, include the loader, the preprocessing pipeline, the schema, and summary statistics, and state access terms in the README.
- Never bypass a data-use agreement to help reviewers; explain the constraint instead. A stated limitation is a clarity point, a violated license is misconduct under the AUAI code of conduct.
Licensing notes
- The review-time ZIP needs no license drama, but the post-acceptance release does: choose deliberately between permissive (MIT/BSD/Apache-2.0) and copyleft, and check employer policy before the camera-ready deadline forces a rushed default.
- Third-party code vendored into the archive keeps its own license; verify compatibility now, since reviewers occasionally notice a GPL file inside an "MIT" archive and read it as carelessness.
- Data licensing is separate from code licensing; a permissive code license does not launder a restricted dataset.
After acceptance
Convert the anonymous ZIP into a public artifact worth citing: a tagged repository, an archival DOI where your institution supports one, a license chosen deliberately, and the README rewritten from "reviewer instructions" to "user instructions." Link it from the camera-ready — post-acceptance is when the CFP's release encouragement costs you nothing and earns citations.
Output format
[Artifact scope] supplement ZIP / public release / both
[Five-minute path] <command a reviewer runs first, and its runtime>
[Claim coverage] <headline results reproducible / total>
[Anonymity scan] clean / leaks: <files>
[Size] <ZIP size vs current cap>
[Data strategy] shipped / generated / loader-plus-terms