# Ecai Artifact Evaluation > Use when planning the reproducibility/artifact story for an ECAI paper — noting that ECAI/FAIA has NO ACM/IEEE-style artifact-badging committee, so credibility is carried by the paper and its supplement and judged by the same reviewers, and adapting an ML-style reproducibility package (or a complete proof appendix for theory work) to that reality. - Skill: `thedixitjain/ecai-artifact-evaluation` (Agent Skill) - Install (CLI): `npx skillmds add thedixitjain/ecai-artifact-evaluation` - Raw SKILL.md: https://api.skillmd.com/api/skills/thedixitjain/ecai-artifact-evaluation/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: thedixitjain (https://skillmd.com/u/thedixitjain) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/thedixitjain/ecai-artifact-evaluation --- # ECAI Artifact Evaluation Start with a correction that saves authors from importing the wrong workflow: **ECAI does not run an ACM/IEEE-style artifact-evaluation track with a separate badge committee.** There is no "Artifacts Available / Functional / Reusable / Reproduced" pipeline as at ACM SIGSOFT venues, and no separate artifact deadline to hit after acceptance. In ECAI, the reproducibility story is **carried by the paper and its supplement and judged by the same reviewers** who read the paper, during the one review round. That makes the "artifact" a **submission-time asset**, not a post-acceptance badge chase. Its job is to make the reviewer trust the claim inside a 7-page body. (Confirm on the current call whether the edition adds any optional reproducibility checklist or appendix mechanism — this is **待核实** per cycle and can differ between a standalone ECAI and the joint IJCAI-ECAI 2026.) ## Match the artifact to the contribution shape ECAI is a **general-AI** venue, so "artifact" means different things: | Contribution shape | The credibility artifact is... | |---|---| | Theory / KR / argumentation | A **complete proof appendix** (full proofs the body only sketches) plus, if applicable, a reference solver/encoding | | Planning / search / optimization | The **domain files, instances, seeds**, and a runnable implementation reproducing the reported node/quality numbers | | Machine learning | Code, data (or a loader), configs, seeds, and **cached outputs** so results reproduce without live API calls | | Multi-agent systems | The environment, agent code, and the exact evaluation protocol (episodes, seeds, metrics) | | Applied AI (PAIS) | Enough of the pipeline and (sanitized) data to make the **deployment claim** credible | ## What "good" looks like at review time - **Anonymized.** The supplement is read under double-blind review; strip repository owners, institution names, and system names that identify you (`ecai-submission`). - **Self-contained.** A reviewer opens it once, in a short window; it must run or be readable without chasing dependencies or your lab's private data. - **Decision-critical content stays in the body.** The supplement holds *support* (full proofs, extra tables, code) — not the claim itself. Nothing a reviewer needs to *judge* the paper may live only in the supplement (`ecai-supplementary`). - **Proportional.** Match effort to the claim: a theory paper's artifact is a rigorous proof appendix, not a Docker image; an empirical paper's artifact is a runnable, seeded package. ## A pragmatic checklist (adapt, don't badge-chase) ```text [ ] Full proofs present for every theorem the body sketches (theory work) [ ] Code runs from a clean checkout with a documented entrypoint (empirical work) [ ] Data included or a script fetches a versioned public source; seeds fixed [ ] Cached model/API outputs included so results do not re-sample at run time [ ] A short README maps each paper claim/table -> the file that reproduces it [ ] Archive anonymized: no owner, institution, funding, or system-name leaks [ ] Total size and runtime reasonable for a reviewer's one-pass read ``` ## Do not import the wrong machinery - **No ACM/IEEE badges.** Do not promise "Artifacts Evaluated - Reusable" or design around a badge committee — none exists at ECAI. Credibility is reviewer-judged, in-band. - **No separate artifact-track deadline.** Everything ships with the paper (abstract 12 Jan / paper 19 Jan for IJCAI-ECAI 2026); there is no later artifact submission. - **Not a leaderboard.** ECAI values understanding (a proof, a fair comparison) over a single benchmark number; an artifact that only re-prints a leaderboard score misses the venue's bar (`ecai-experiments`). ## Post-acceptance: make it permanent and open Once accepted, convert the anonymized supplement into a **permanent, open** release to match ECAI's open-access ethos: - Deposit code/data in a DOI-issuing archive (e.g. Zenodo/Software Heritage) with an open license. - De-anonymize repository owners and restore acknowledgements (`ecai-camera-ready`). - Put the permanent link in the camera-ready so the open-access paper points to a stable artifact. ## Output format ```text [Artifact type] proof appendix / runnable code+data / environment+protocol / deployment pipeline [Anonymity] clean / leaks: [Claim map] each theorem/table -> proof or reproducing file [Self-containment] runs/readable in one pass? missing deps: [Reality check] no ACM/IEEE badge assumed; nothing decision-critical hidden in the supplement [Post-acceptance] DOI archive + open license + de-anonymized link planned for camera-ready ``` --- **Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `ECAI-Skills/skills/ecai-artifact-evaluation/SKILL.md`