Ecai Reproducibility

ECAI Reproducibility

ECAI reproducibility is in-band: there is no separate artifact-evaluation track, so the same reviewers who judge the paper judge whether the results and proofs are believable, from the 7-page body plus an anonymized supplement. Build the reproducibility story to survive that read — one pass, double-blind, in a short window — not a badge committee.

Because ECAI is a general-AI venue, "reproducible" means different things across its breadth. Pick the mode that matches your contribution.

Mode 1 — Theory / KR / argumentation: proofs are the artifact

• The body sketches; the supplement carries every full proof. A theorem stated without a checkable proof is a claim, not a result.
• State all assumptions explicitly (finiteness, admissibility, monotonicity, language fragment). The most common reject-driving misreading is a reviewer assuming a hidden condition.
• If the theory has a computational side (a solver, an encoding, complexity results), include a reference implementation or the exact encoding so a reviewer can re-run a small instance.
• Define objects once, precisely; ECAI's symbolic-AI reviewers check definitions against lemmas.

Mode 2 — Empirical / ML / planning: seed, cache, pin

• Fix and report seeds; report central tendency and spread across seeds, not a single lucky run (ecai-experiments).
• Cache raw outputs (model predictions, planner traces, API responses) so results reproduce without live calls — a package that re-queries an API re-samples rather than reproduces.
• Pin provenance: dataset name and version/date, preprocessing scripts, model identifiers with dates, hardware where it affects timing.
• Provide a claim→file map: each reported table/number points to the script that regenerates it.

Provenance pinning (both modes, where applicable)

[ ] Dataset: name, version/DOI, download date, license, preprocessing script committed
[ ] Splits: exact train/val/test (or instance sets) fixed and included or scripted
[ ] Models: identifiers + dates (for hosted/LLM components); prompts/configs committed
[ ] Seeds: fixed and reported; number of runs stated
[ ] Environment: dependency versions pinned (lockfile / environment.yml / requirements)
[ ] Outputs: raw results cached so re-run does not depend on a live service

Double-blind, in the supplement too

The supplement is read under double-blind review. Anonymize it as carefully as the PDF:

# Sweep the staged supplement before zipping
grep -rniE 'university|@[a-z0-9.]+\.(edu|ac\.[a-z]+)|acknowledg|funded by|grant (no|number)' supplement/ | head
unzip -l supplement.zip | grep -Ei '\.git/|/home/|/Users/|\.DS_Store' | head

Strip repository owners, institution names, funding lines, and any system named after your group. A de-anonymizing supplement can trigger a summary reject before the science is even read.

Honesty over completeness

• If data cannot be shared (privacy, licensing, industrial confidentiality — common in PAIS applications), say so and why, and share what you can (code, a synthetic sample, the protocol). A silent gap reads worse than a stated, justified limitation.
• Do not claim reproducibility you have not tested. Run the package from a clean checkout yourself before submitting.

Fit the 7-page body

Reproducibility content that a reviewer needs to judge the paper (the core proof idea, the evaluation protocol, the key numbers) belongs in the body; full proofs, extra tables, and code belong in the supplement. Nothing decision-critical may live only outside the 7 pages (ecai-supplementary).

Post-acceptance

Convert the anonymized supplement into a permanent, open release — DOI-issuing archive, open license, de-anonymized owners — and link it from the open-access camera-ready (ecai-camera-ready).

Output format

[Mode] theory (proof appendix) / empirical (seeded+cached) / mixed
[Proof completeness] every theorem has a full checkable proof + explicit assumptions? yes/no
[Provenance] datasets/models/seeds/env pinned? gaps: <list>
[Claim map] each table/number -> regenerating file
[Anonymity] supplement clean / leaks: <where>
[Body/supplement split] nothing decision-critical outside the 7-page body
[Post-acceptance] DOI + open license + de-anonymized link planned

Source: brycewang-stanford/Awesome-Journal-Skills → ECAI-Skills/skills/ecai-reproducibility/SKILL.md