thedixitjain ASE Artifact Evaluation
Convert the accepted paper's package into badges. ASE runs an Artifact Evaluation track offering the Artifacts Available and Artifacts Reusable badges (ACM scheme). Because ASE proceedings are indexed in both IEEE Xplore and the ACM Digital Library, an earned badge appears on the paper's front page in both. Evaluation happens on the track's own deadline, separate from the research-track notification — stage the package before then.
The two badges (verify the current call)
- Artifacts Available — the artifact is placed in a publicly accessible archival repository with a DOI (Zenodo, figshare, Software Heritage, or an institutional/ACM repository). A personal GitHub link alone is not archival; mint a DOI.
- Artifacts Reusable — the artifact significantly exceeds minimal functionality: it is carefully documented and well-structured so a third party can reuse the tool, not merely reproduce your tables. This is the higher bar and where automated-SE tools usually need the most work.
- Whether Functional and Results Reproduced badges are also offered at a given edition is 待核实 — confirm on the current Artifact Evaluation call.
From the submission artifact to the badge artifact
The review-time (anonymized) artifact and the badge artifact are the same package matured. After
acceptance you can de-anonymize it, but the substance should already be there if you followed
ase-reproducibility.
[De-anonymize] restore the real tool name, authors, repository, license.
[Archive] deposit in a DOI-issuing archive; the DOI is what "Available" certifies.
[Document] README with exact run path, expected outputs, and a small worked example.
[Environment] container/lockfile pinning deps + the exact tool commit; note hardware needs.
[Reuse story] show how to run the tool on a NEW input, not just replay your experiments.
Reusable is about strangers, not your tables
Evaluators judge reusability, so write for someone who wants to use your automation on their own code:
- A clear entry point and documented inputs/outputs.
- Instructions to run on a new subject, with a template config.
- Sensible structure (source vs. data vs. scripts), an open license, and dependency pinning.
- Removal of dead scripts, secrets, and machine-specific paths.
Evaluator-proofing checklist
[Runs clean] fresh environment (container) -> documented command -> expected output, no manual patching
[DOI] archival deposit with a DOI + open license (for Available)
[Docs] README covers install, run, expected results, and reuse on a new input (for Reusable)
[Provenance] subject SHAs, dataset version, seeds, model IDs/dates + cached outputs included
[Scope honesty] hardware/time requirements and known limitations stated up front
[No secrets] API keys, tokens, private paths removed
Timing and scope
- The Artifact Evaluation deadline follows research-track acceptance; treat it as a real milestone, not an afterthought — a strong tool with a weak package earns no badge.
- Evaluators are often students and junior researchers on a schedule: an artifact that needs a live API key, unpinned dependencies, or your specific cluster will fail on setup regardless of the underlying quality.
- Badges are recognition, not re-review of the science; the paper is already accepted. The goal is durable, reusable automation.
Output format
[Target badges] Available / Reusable (Functional/Reproduced 待核实 for this edition)
[Archive] DOI minted? open license?
[Runs clean] fresh-env command -> expected output, no manual fixes?
[Reusable] docs + run-on-new-input path present?
[Provenance] SHAs / dataset version / seeds / model IDs / cached outputs bundled?
[Blockers] <ordered fixes before the AE deadline>
Source: brycewang-stanford/Awesome-Journal-Skills → ASE-Skills/skills/ase-artifact-evaluation/SKILL.md