brycewang-stanford SoCC Artifact Evaluation
Use this for artifact preparation. SoCC — as an ACM venue — follows the ACM Artifact Review and Badging scheme where an edition offers evaluation. First, verify whether the current SoCC edition runs a dedicated artifact-evaluation track, and which badges it offers — unlike some sibling systems flagships that run a standing AE process, SoCC's artifact track and badge set are decided per edition and are 待核实 as of 2026-07-09. The advice below applies once the edition's call confirms evaluation.
Two things to internalize: badges are earned by evaluators actually reproducing your cloud results, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived).
The ACM badges (verify the current set and names)
| Badge | What it certifies | What earns it for a cloud paper |
|---|---|---|
| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |
| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | A documented testbed setup, a workload replay, and expected outputs |
| Artifacts Evaluated - Reusable | Others can build on it | Functional plus careful docs, structure, licensing, and a portable harness |
| Results Reproduced | An evaluator reproduced the paper's key results | A turnkey path from the artifact to the headline throughput/tail/cost numbers |
Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced require the evaluator's own run to succeed, so the failure mode is always "did not run on their testbed," never "the idea was weak."
What a cloud-systems evaluator opens first
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| A cloud system/mechanism | The README and one setup+run command | Undocumented cluster assumptions; only-runs-on-authors'-testbed |
| A measurement/trace study | The scripts that turn the trace into the paper's figures | Numbers in the PDF no script reproduces; trace missing |
| A scheduling/serverless result | The workload generator + the tail/cost measurement scripts | Only the mean reproduces; tail and cost cannot be regenerated |
| A large-scale deployment | A scaled-down but faithful reproduction path | Requires a proprietary cluster; no smaller-scale replay |
Assume an evaluator has a bounded time budget and cannot reserve your 200-node cluster. Provide a scaled-down reproduction that still regenerates the shape of the tail and cost results, plus a clear statement of what needs full scale.
Packaging plan
[Environment] ship a Dockerfile / pinned environment AND a testbed description (node counts,
instance types, OS, kernel) so a run is reproducible
[Workloads] the workload generator or the (anonymized, then released) trace, not just a pointer
[README] one-screen orientation: what it is, how to set up, how to run a small demo, how to
reproduce each figure, expected runtime and outputs
[Mapping] an explicit table: paper claim -> script -> expected result (incl. tail and cost)
[Scaled path] a small-scale reproduction that runs without the full cluster
[Provenance] commit SHAs, trace extraction dates, instance types, seeds, run counts
[License] an OSI-approved license so the artifact can be badged Reusable
[Archive] deposit in a DOI-issuing repository for the Available badge
Anonymized review artifact vs. badge artifact
- At submission: the artifact is anonymized for the paper's reviewers — no cluster names, provider hints, owner strings, or identity-revealing trace provenance, and no live repository that discloses authors.
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version any artifact evaluators badge and the camera-ready cites.
Worked vignette: packaging an autoscaler + trace study
A paper contributes a tail-aware autoscaler and a measurement of the cost-tail gap. To target
Reusable and Reproduced: ship a Docker image with the controller pre-built; a run_demo.sh that
replays a short trace slice on a few nodes in minutes and prints p99 and instance-seconds; a
reproduce/ directory whose scripts regenerate each figure from logged runs; a claim-to-script
mapping table in the README; the (released) trace-replay harness with pinned SHAs; and an
MIT/Apache license. State honestly which figures are turnkey at small scale and which need the full
testbed.
Calibration
- Confirm the track exists for this edition before planning; SoCC's AE track and badge set are 待核实 per cycle.
- Reproducing tail and cost, not just the mean, is the cloud-specific bar; design the package so an evaluator can regenerate them.
- Badge names, the exact set offered, and whether evaluation is single- or double-anonymous vary by edition — confirm on the current call.
Output format
[Track status] SoCC AE track confirmed for this edition? yes/no/待核实
[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <system/workloads/trace/scripts/testbed-desc/provenance/license>
[Small-scale test] does setup + demo reproduce tail+cost shape without the full cluster? yes/no
[Claim mapping] <claim -> script -> expected result (incl. tail/cost) present? yes/no>
[Fixes before upload] <ordered list>