thedixitjain FSE Artifact Evaluation
Use this for the artifact track. FSE follows the ACM Artifact Review and Badging scheme, and the artifact evaluation is a separate, post-acceptance process with its own deadline. Two things to internalize: badges are earned by evaluators actually using your package, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived).
The ACM badges (verify the current set and names)
| Badge | What it certifies | What earns it |
|---|---|---|
| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |
| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | A clean-machine install, a demo, and documented expected outputs |
| Artifacts Evaluated - Reusable | Others can build on it | The Functional bar plus careful docs, structure, and licensing |
| Results Reproduced | An evaluator reproduced the paper's key results | A turnkey path from the artifact to the headline numbers |
Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced require the evaluator's own run to succeed, so the failure mode is always "did not run on their machine," never "the idea was weak."
What SIGSOFT evaluators open first
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| A tool/technique | The README and one install/run command | Undocumented dependencies; only-works-on-authors'-laptop |
| An empirical study | The scripts that turn data into the paper's tables | Numbers in the PDF that no script reproduces |
| A mined dataset | The extraction scripts + the extracted data | Query shipped, data missing; provenance unpinned |
| An LLM-based result | Cached prompts/outputs + model IDs | Requires live API keys; not reproducible |
Assume an evaluator gives your package a bounded time budget on a clean machine. Design for the first ten minutes to succeed.
Packaging plan
[Container] ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
"install these 40 things by hand"
[README] one-screen orientation: what it is, how to install, how to run the demo, how to
reproduce each claim, expected runtime and outputs
[Mapping] an explicit table: paper claim -> script -> expected result
[Data] the extracted dataset itself (or documented access), not just the query
[Provenance] repo SHAs, extraction dates, model IDs/dates, seeds
[License] an OSI-approved license so the artifact can be badged Reusable
[Archive] deposit in a DOI-issuing repository for the Available badge
Anonymized review artifact vs. badge artifact
- At submission: the artifact is anonymized for the paper's reviewers — no owner strings, cluster paths, lab names, or identity-revealing links, and no live repository that discloses authors.
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version the artifact evaluators badge and the camera-ready cites.
Worked vignette: packaging a detection tool + study
A paper contributes a defect-detection tool and an empirical evaluation. To target Reusable and
Reproduced: ship a Docker image with the tool pre-built; a run_demo.sh that detects on a small
bundled project in under a minute; a reproduce/ directory whose scripts regenerate each table
from logged results; a claim-to-script mapping table in the README; the extracted evaluation
dataset with pinned SHAs; and an MIT/Apache license. State honestly which results are turnkey and
which need the full (slow) dataset run.
Calibration
- The artifact deadline is after acceptance and independent of the camera-ready; do not conflate them.
- Badge names, the exact set offered, and whether evaluation is single- or double-anonymous vary by cycle — confirm on the current artifact-track call.
Output format
[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <tool/data/scripts/provenance/license>
[Ten-minute test] does install + demo succeed on a clean machine? yes/no
[Claim mapping] <claim -> script -> expected result present? yes/no>
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → FSE-Skills/skills/fse-artifact-evaluation/SKILL.md