brycewang-stanford SIGCOMM Artifact Evaluation
Use this for the post-acceptance artifact submission. SIGCOMM helped establish artifact evaluation in networking, and the venue's "runnable papers" culture sets a high bar: a committee attempts to build and, where a badge requires it, reproduce your headline result. Reopen the current Call for Artifacts for this edition's exact badge set and dates (待核实 for 2026).
Badge selection is claim calibration
Choose the badges you can honestly support; the badge is a promise about what a stranger can verify, not a participation ribbon.
| ACM badge | What it promises | What the AEC will do |
|---|---|---|
| Artifacts Available | Permanent public retrieval via an archival DOI | Confirm the archive resolves and is complete |
| Artifacts Evaluated (Functional) | The artifact builds and runs as documented | Follow your README end to end on their setup |
| Results Reproduced | The paper's main results follow from the artifact | Re-run to obtain the headline numbers within tolerance |
Aim the highest badge at the result you most want believed, and be candid where hardware or scale makes full reproduction infeasible.
The networking-specific hard part
Generic artifact tooling cannot judge what SIGCOMM evaluators care about most: a result measured on a rack of switches or a wide-area testbed rarely rebuilds on a laptop. Plan for the gap:
- Downscale the topology. Ship a small emulated or Mininet-style topology that reproduces the mechanism's behavior, with a documented map from the downscaled run to the paper's full-scale figure.
- Substitute the trace legally. If the production trace cannot ship, provide a synthetic or public-trace substitute with the same statistical character, and state exactly which figures used which input.
- Pin the environment. Kernel and NIC features, switch firmware or programmable-data-plane toolchain versions, and clock/timestamp sources all move results; record them.
- Expose the run count. Every reported percentile needs its replication count and the seed or workload driver, or the tail cannot be reproduced.
What evaluators open first
artifact/
README # 5-minute smoke run FIRST, then the full path
smoke/ # one command -> one small figure that proves it runs
topology/ # downscaled + a map to the paper's full-scale setup
traces/ # substitute inputs + provenance + which-figure-uses-what
configs/ # exact parameters per experiment
scripts/ # regenerate each figure from logged results
results/ # logged raw outputs the plots are built from
ENVIRONMENT.md # kernel/NIC/switch/toolchain versions, hardware
Assume the evaluator runs the smoke path first and stops if it fails, so make one command produce one convincing small figure before anything else is polished.
Vignette: packaging a data-center transport result
A paper reports a tail-FCT win on a 128-server testbed. Turnkey plan: a Mininet topology that reproduces the incast dynamic at small scale; a synthetic RPC workload matching the trace's flow-size distribution; a driver that emits the FCT distribution over 20 replays; and a script that plots the 99th percentile directly from logged results so the artifact figure and the paper figure cannot diverge. The README states plainly which numbers are full-scale (paper) and which are downscaled (artifact).
Output format
[Badges sought] available / functional / reproduced — each justified
[Smoke path] one command -> one figure? yes/no
[Topology] full-scale -> downscaled map documented
[Trace handling] shipped / substituted (+ provenance) / described
[Environment] kernel/NIC/switch/toolchain pinned
[Fixes before AEC upload] <ordered list>