brycewang-stanford SIGMETRICS Artifact Evaluation
Use this for artifact/reproducibility packaging. SIGMETRICS sits in the ACM ecosystem and, where an artifact track runs, follows the ACM Artifact Review and Badging scheme. Two things to internalize: badges are earned by evaluators actually using your package, and — distinctively for SIGMETRICS — the package usually has to let an evaluator regenerate the figures from a seeded simulation and see them match the analytic prediction, not only run a tool. Confirm on the current cycle whether a formal artifact track exists and its timing (待核实).
The ACM badges (verify the current set and names)
| Badge | What it certifies | What earns it |
|---|---|---|
| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |
| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | A clean-machine install, a demo run of the simulator, documented expected outputs |
| Artifacts Evaluated - Reusable | Others can build on it | The Functional bar plus careful docs, structure, and licensing |
| Results Reproduced | An evaluator reproduced the paper's key results | A turnkey path from the artifact to the headline figures/numbers (and the analytic overlay) |
Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced require the evaluator's own run to succeed, so the failure mode is "did not run / figures did not match on their machine," never "the theorem was weak."
What performance-evaluation evaluators open first
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| An analytic bound + simulation | The script that regenerates the analysis-vs-simulation figure | Figure hard-coded; simulator does not actually produce the plotted curve |
| A measurement study | The scripts that turn the trace into the paper's tables | Numbers in the PDF that no script reproduces; trace missing |
| A scheduling/queueing policy | The seeded simulator and its steady-state handling | Non-deterministic runs; no seeds; warm-up not handled |
| A learning-for-systems result | Code plotting empirical regret against the proven bound | Requires unavailable data; guarantee not empirically checked |
Assume an evaluator gives your package a bounded time budget on a clean machine. Design for the first ten minutes to succeed: a small demo that regenerates one headline figure quickly.
Packaging plan
[Container] ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
"install these 40 things by hand"
[README] one-screen orientation: what the model is, how to install, how to run the simulator
demo, how to regenerate each figure, expected runtime and outputs
[Mapping] an explicit table: paper claim/figure -> script -> expected result (incl. the
analytic overlay it should match)
[Simulator] seeded, steady-state-aware; a fast demo config and the full (slow) config
[Data] the processed trace/dataset (or documented access), not just the collection query
[Proofs] the derivation appendix, so the analytic side is checkable alongside the code
[License] an OSI-approved license so the artifact can be badged Reusable
[Archive] deposit in a DOI-issuing repository for the Available badge
Anonymized review artifact vs. badge artifact
- At submission: the artifact is anonymized for the paper's reviewers — no owner strings, cluster paths, group names, or identity-revealing trace provenance (Operational Systems Track excepted).
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version the badges attach to and the POMACS camera-ready cites.
Worked vignette: packaging a scheduling-policy paper
A paper contributes a scheduling theorem and a trace-driven evaluation. To target Reusable and
Reproduced: ship a Docker image with the simulator pre-built; a run_demo.sh that regenerates the
analysis-vs-simulation figure on a small config in under a minute; a reproduce/ directory whose
scripts regenerate each table and the trace-driven comparison from logged runs; a
claim-to-figure-to-script mapping in the README; the processed trace with pinned provenance; the
proof appendix; and an MIT/Apache license. State honestly which figures are turnkey and which need
the full (slow) simulation sweep.
Calibration
- Whether an artifact/reproducibility track runs a given cycle, its badges, timing, and whether it is mandatory or optional — all cycle-volatile; confirm on the current call (待核实).
- For POMACS, badges are typically pursued around/after acceptance; do not conflate the artifact process with the paper's shepherding.
Output format
[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <model/simulator/trace/proofs/provenance/license>
[Ten-minute test] does install + demo regenerate one headline figure on a clean machine? yes/no
[Claim mapping] <claim/figure -> script -> expected result (matches analysis?) present? yes/no>
[Fixes before upload] <ordered list>