Neurips Artifact Evaluation

NeurIPS Artifact Evaluation

NeurIPS main track does not reduce artifact quality to a generic badge workflow. It expects code, data, and execution details when they are needed to support the scientific claim, and its checklist and code/data guidance make artifact quality visible to reviewers.

Artifact decision

• If the contribution is a method, include training and evaluation code or justify why it cannot be shared.
• If the contribution is a dataset or benchmark, provide metadata, license, preservation plan, representative-use discussion, and access restrictions.
• If the contribution depends on a model, include weights, prompts, decoding settings, compute resources, or a precise explanation of unavailable components.
• If the contribution is theoretical, artifact focus may shift to proof checks, symbolic scripts, experiment notebooks, or counterexample generation.

Anonymous review package

• Keep the ZIP within the current official size limit and anonymize filenames, repository URLs, usernames, commit history, model cards, dataset cards, comments, notebooks, and logs.
• Include a short README with exact commands, environment, expected runtime, hardware assumptions, and which experiments are reproducible from the package.
• Do not require reviewers to run unsafe code outside a secure environment.
• Avoid external links unless the current policy allows them and anonymous browsing is guaranteed.

Public release package

• De-anonymize accepted artifacts.
• Add licenses for code, data, model weights, and generated outputs.
• Archive code in a durable service when appropriate; NeurIPS MLRC guidance recommends Software Heritage for reproducibility papers.
• Keep a mapping from paper claims to commands or notebooks so users can reproduce headline results.

Output format

[Artifact role] method / dataset / benchmark / model / demo / proof / none
[Review package] sufficient / insufficient
[Anonymity risks] <paths, metadata, URLs, usernames>
[Reproducibility gaps] <commands, environment, data, hardware, licenses>
[Public-release plan] <archive, DOI, license, docs>

Source: brycewang-stanford/Awesome-Journal-Skills → NeurIPS-Skills/skills/neurips-artifact-evaluation/SKILL.md