brycewang-stanford ICSE Artifact Evaluation
ICSE's artifact track is a post-acceptance, opt-in evaluation: once the
paper is in, you submit the artifact to a separate committee that awards ACM-
standard badges printed on the published paper. It is the formal payoff of the
verifiability work done at review time — if icse-reproducibility was
followed, this skill is mostly packaging and paperwork.
The badge system, as ICSE runs it
Verified against the ICSE 2026 artifact-evaluation track (read 2026-07-08; ICSE follows the ACM artifact review and badging standard):
| Badge | Who applies | What evaluators check |
|---|---|---|
| Artifacts Available | Authors of papers accepted to Research, SEIP, NIER, SEIS, Doctoral Symposium, Demonstrations | Artifact is permanently archived in a public, DOI-issuing repository — not a personal GitHub, which can be deleted or rewritten |
| Artifacts Reusable (Evaluated) | Same tracks | Quality "significantly exceeds minimal functionality": careful documentation, structure that supports reuse and repurposing by others |
| Results Reproduced / Replicated | Authors of prior published SE work | Independent obtainment of the paper's results, per the ACM definitions |
Two structural notes. First, Available is about archival, Reusable about quality — a beautifully engineered artifact on a transient URL fails the first, a DOI'd tarball nobody can run fails the second; aim for both. Second, the Reproduced/Replicated badges are a lane for already-published work, which is also how your artifact gets used: build it expecting a stranger to attempt exactly that next year.
Timeline anchor
ICSE 2026 ran artifact registration on January 8 and submission on January 15, 2026 — weeks after final paper decisions (December 18 in the 2027-cycle calendar) and months before the conference. The 2027 dates were unposted at check time (待核实 at the ICSE 2027 artifact-evaluation page). Planning consequence: the artifact deadline lands in the first working days of January, so the package must be effectively done before the holidays.
From review package to badge submission
The anonymized review-time package needs five upgrades:
- De-anonymize deliberately — restore names, affiliations, the real license holder, and the citation file; re-point links from the anonymizing host to the permanent home.
- Archive with a DOI — deposit in a DOI-issuing archive (Zenodo-class,
institutional repository); record the DOI in the paper's availability
statement before the camera-ready freezes (
icse-camera-ready). - License openly — the ICSE AE guidance encourages open-source licenses (MIT, Apache-2.0, GPL) for code and open-data licenses (CC-BY, CC0) for data; a package with no license is legally unusable and evaluators notice.
- Write for the evaluator's clock — a stranger, on their own machine, with a review load. Provide a container or exact environment recipe, a smoke test in minutes, and the full runs' costs stated honestly up front.
- Document for reuse, not just replay — the Reusable bar is repurposing: how to run on new subjects, extend the benchmark, or swap the model — one section each in the README.
The evaluator-proof README
# Artifact for "<paper title>" (ICSE 20XX, paper #NNN)
## Claims supported
Table 3 (RQ1), Fig. 5 (RQ2) fully reproducible below.
Table 6 requires 40 GPU-hours; cached outputs + verification script provided.
## Kick the tires (10 min, laptop)
docker load < artifact.tar && ./smoke_test.sh # expected output shown below
## Full reproduction
./reproduce_all.sh # ~6 h CPU; per-table scripts listed in scripts/
## Reusing beyond the paper
- New subject program: docs/add_subject.md
- Different model backend: docs/swap_model.md
## Requirements / License / Citation
The claims-supported section is the trust anchor: state plainly which paper results the artifact does and does not cover, and give a cached-output verification path for anything expensive. Evaluators reward honesty about coverage and punish discovering gaps themselves.
Scoping what the artifact claims
Badge submissions fail more often from over-claiming than under-building. Declare a claims scope narrower than the paper if that is the truth: "the artifact reproduces RQ1 and RQ2; RQ3's industrial dataset is excluded under NDA, and we include the analysis scripts plus synthetic sample data so the pipeline itself is checkable." Evaluators can only award what they can verify, and a scoped-honest artifact that fully delivers its scope reads better than an everything-artifact with two broken paths. Match the scope statement to the paper's Data Availability section — a mismatch between the two is the first thing a careful evaluator diffs.
Failure modes that cost badges
- Bit-rot between acceptance and January: dependency drift breaks the build; freeze a container image the week of acceptance.
- Undeclared network/credential needs: an artifact that must call a paid API mid-evaluation stalls; cache responses and make live calls optional.
- Doc-code mismatch after camera-ready edits changed table numbers — re-run the claims map against the final paper, not the submission.
- Repository-instead-of-archive: linking GitHub
mainwhere the badge requires an immutable DOI deposit.
Reverify each cycle
Track list, badge set, deadlines, and submission form are set by each year's AE chairs; the ACM badging vocabulary is the stable layer. Confirm the 2027 call before promising badge scope to coauthors — and note the badge is opt-in: absence of one on a published ICSE paper signals nothing about its authors, only about their January.
Output format
[Badges targeted] Available / Reusable / (Reproduced-Replicated for prior work)
[Archival] DOI minted? license file? citation file?
[Evaluator path] smoke-test minutes; full-run cost; cached-output coverage
[Reuse docs] new-subject and extension guides present?
[Deadline plan] frozen container by <date>; submission by the January window