IJCAI Artifact Evaluation
Use this for artifact packaging around IJCAI. Treat the current reproducibility guidelines
and supplementary-material rules as controlling; do not assume a separate formal artifact
evaluation track unless the current cycle announces one.
Package design
- Decide what reviewers need to classify the results as convincing or credible: proofs,
pseudocode, datasets, code, model cards, logs, environment details, or ablation notebooks.
- Keep essential evidence in the paper whenever space allows because reviewers are not
required to read supplementary material.
- Put optional evidence in the Technical Appendix or ZIP, respecting the current size and
format limit. IJCAI-ECAI 2026 allowed up to 50MB in PDF or ZIP form.
- Anonymize repository paths, user names, institutions, license headers, model checkpoints,
data provenance, and notebook metadata.
- Include a minimal run map: environment, dependencies, hardware, commands, expected outputs,
runtime, seeds, and known limitations.
- For proprietary or restricted data/code, explain why it cannot be shared and provide enough
detail for in-principle reproduction.
Evidence by claim type
IJCAI usually has no separate artifact-evaluation badge, so the artifact's job is to move the
reproducibility rating toward convincing and to pre-empt a broad PC's doubt. Match the package
to the claim.
| Claim |
Artifact that convinces an IJCAI reviewer |
Weak substitute to avoid |
| New search/planning algorithm |
Runnable solver, instance generator, seeds, time/memory limits |
A results CSV with no way to regenerate it |
| Theoretical guarantee |
Full proofs, assumption list, citations to formal tools |
"Proof omitted for space" with no appendix |
| Multi-agent protocol |
Simulator, opponent policies, randomization seeds |
Screenshots of one run |
| Learning result |
Code, environment file, configs, compute and runtime |
Final-number table only |
| Dataset |
Datasheet, license, controlled-access path |
Vague "available on request" |
Worked vignette: packaging a SAT-solver paper
A SAT/CSP paper claims a new restart heuristic wins on hard industrial instances. Strong
package: the solver binary or source, the exact instance set or a deterministic generator,
solver and compiler versions, per-instance runtimes with the timeout stated, and a run map
showing one command that reproduces the cactus plot. Anonymize the repository name, license
headers, and any cluster paths. This lets a skeptical constraint-reasoning reviewer re-run a
sample and raises the rating from credible to convincing without needing a badge track.
Reviewer pushback and the venue-specific fix
- "Cannot regenerate the benchmark instances." Ship a generator plus seeds, not just outputs.
- "Artifact leaks author identity." Scrub paths, license headers, checkpoint names, and
notebook metadata before the full-paper deadline; there is no late re-upload.
- "Restricted data blocks reproduction." Document the legal barrier and give enough protocol
for in-principle reproduction rather than implying a release you cannot deliver.
Output format
[Artifact role] paper evidence / supplement / post-acceptance release
[Contents] <proofs/code/data/models/logs/docs>
[Anonymity risks] <paths/licenses/metadata/URLs>
[Reproducibility claim] convincing / credible / weak
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → IJCAI-Skills/skills/ijcai-artifact-evaluation/SKILL.md
1---2name: ijcai-artifact-evaluation3description: Use when packaging IJCAI or IJCAI-ECAI code, data, proofs, models, and appendices as reproducibility evidence or supplementary material, especially when there is no separate artifact-evaluation badge but reviewers need convincing, anonymous, deadline-safe evidence.4---567# IJCAI Artifact Evaluation89Use this for artifact packaging around IJCAI. Treat the current reproducibility guidelines10and supplementary-material rules as controlling; do not assume a separate formal artifact11evaluation track unless the current cycle announces one.1213## Package design1415- Decide what reviewers need to classify the results as convincing or credible: proofs,16 pseudocode, datasets, code, model cards, logs, environment details, or ablation notebooks.17- Keep essential evidence in the paper whenever space allows because reviewers are not18 required to read supplementary material.19- Put optional evidence in the Technical Appendix or ZIP, respecting the current size and20 format limit. IJCAI-ECAI 2026 allowed up to 50MB in PDF or ZIP form.21- Anonymize repository paths, user names, institutions, license headers, model checkpoints,22 data provenance, and notebook metadata.23- Include a minimal run map: environment, dependencies, hardware, commands, expected outputs,24 runtime, seeds, and known limitations.25- For proprietary or restricted data/code, explain why it cannot be shared and provide enough26 detail for in-principle reproduction.2728## Evidence by claim type2930IJCAI usually has no separate artifact-evaluation badge, so the artifact's job is to move the31reproducibility rating toward convincing and to pre-empt a broad PC's doubt. Match the package32to the claim.3334| Claim | Artifact that convinces an IJCAI reviewer | Weak substitute to avoid |35| --- | --- | --- |36| New search/planning algorithm | Runnable solver, instance generator, seeds, time/memory limits | A results CSV with no way to regenerate it |37| Theoretical guarantee | Full proofs, assumption list, citations to formal tools | "Proof omitted for space" with no appendix |38| Multi-agent protocol | Simulator, opponent policies, randomization seeds | Screenshots of one run |39| Learning result | Code, environment file, configs, compute and runtime | Final-number table only |40| Dataset | Datasheet, license, controlled-access path | Vague "available on request" |4142## Worked vignette: packaging a SAT-solver paper4344A SAT/CSP paper claims a new restart heuristic wins on hard industrial instances. Strong45package: the solver binary or source, the exact instance set or a deterministic generator,46solver and compiler versions, per-instance runtimes with the timeout stated, and a run map47showing one command that reproduces the cactus plot. Anonymize the repository name, license48headers, and any cluster paths. This lets a skeptical constraint-reasoning reviewer re-run a49sample and raises the rating from credible to convincing without needing a badge track.5051## Reviewer pushback and the venue-specific fix5253- "Cannot regenerate the benchmark instances." Ship a generator plus seeds, not just outputs.54- "Artifact leaks author identity." Scrub paths, license headers, checkpoint names, and55 notebook metadata before the full-paper deadline; there is no late re-upload.56- "Restricted data blocks reproduction." Document the legal barrier and give enough protocol57 for in-principle reproduction rather than implying a release you cannot deliver.5859## Output format6061```text62[Artifact role] paper evidence / supplement / post-acceptance release63[Contents] <proofs/code/data/models/logs/docs>64[Anonymity risks] <paths/licenses/metadata/URLs>65[Reproducibility claim] convincing / credible / weak66[Fixes before upload] <ordered list>67```6869---7071**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `IJCAI-Skills/skills/ijcai-artifact-evaluation/SKILL.md`
thedixitjain