brycewang-stanford SIGMETRICS Reproducibility
Use this before submission and again before the POMACS camera-ready. SIGMETRICS reproducibility has a distinctive shape: the "artifact" is often a proof plus a simulator plus a trace, not only running code. The goal is that a competent reader could re-derive your bound, re-run your simulation to the same curves, and re-analyze your measurement to the same conclusions.
Evidence map
- Map each theorem, bound, and reported number to a verifiable location — a proof in an appendix, a figure regenerated from a seeded simulation, or a script that turns the trace into the table.
- For analytic results, give the full derivation and every assumption; a reader should be able to check the proof and see which assumptions each step uses.
- For simulations, ship a seeded simulator whose scripts regenerate each figure and overlay the analytic prediction, so a reviewer sees model and measurement agree.
- For measurement studies, document the trace source, collection window, sanitization, and the processing scripts; archive the processed dataset or document access.
- Keep the paper and the artifact consistent: a p99 number in the PDF that no simulator run reproduces is the contradiction reviewers read as carelessness.
Reproducibility-claim audit
| Claim in the paper | Weak reproducibility answer | SIGMETRICS-ready answer |
|---|---|---|
| "Theorem 1 bounds the tail" | Proof sketch only | Full proof (appendix) + a simulation that matches the analytic curve |
| "We simulate policy X" | "Simulator available on request" | Seeded simulator + scripts that regenerate each figure from logged runs |
| "We measured system Y" | "Data on request" | Processed dataset (or documented access) + provenance + processing scripts |
| "The learner has low regret" | Empirical curve only | Regret proof + code plotting empirical regret against the bound |
"Available on request" is treated as not available; convert every such line into a concrete, anonymized artifact or an explicit, justified exception (e.g. a proprietary trace, with the methodology fully documented).
Provenance and determinism pinning
[Proof] state every assumption; give the full derivation; note which lemmas each step needs
[Simulation] log seeds; state steady-state/warm-up handling; make figures regenerate deterministically
[Measurement] pin the trace source, collection window, sanitization; archive processed data
[Compute] state hardware, runtime, and number of independent runs so a reader can size a rerun
[Agreement] ship the overlay of analysis vs. simulation so the match is reproducible, not asserted
Degrees of reproducibility (state the one you achieved)
- Turnkey: one documented command regenerates each figure/table from logged simulation runs and reproduces the analytic overlay.
- Scripted: scripts exist but require documented manual steps or access to a restricted trace.
- Descriptive: proofs and methodology detailed enough that a competent reader could rebuild the pipeline.
For SIGMETRICS, aim turnkey for anything a reviewer might rerun quickly (a simulation regenerating a figure, a script producing a table); a proprietary industrial trace may stay scripted with access documented, but the methodology and the analysis code should still be turnkey.
Vignette: a queueing-theory-plus-measurement paper
Consider a paper with a scheduling theorem and a trace-driven evaluation. Its reproducibility spine: the full proof with stated assumptions in an appendix; a seeded simulator whose notebook regenerates the analysis-vs-simulation figure; the trace-processing scripts with pinned provenance; the anonymized processed dataset (or documented access to a restricted one); and the analysis notebooks that turn logged runs into the paper's tables — plus one honest sentence about any assumption that only approximately holds and how §6 bounds it.
Consistency and camera-ready pass
- Before submission: every reported number traces to a proof, a logged simulation run, or a measurement script; the artifact is anonymized (no owner strings, cluster paths, group names).
- Before camera-ready: swap anonymized links for a permanent, DOI-issuing archive, and align the
artifact with any ACM badges you are pursuing (
sigmetrics-artifact-evaluation).
Output format
[Claim inventory] <claim -> proof / simulation run / measurement script>
[Reproducibility] concrete / vague / missing, per claim
[Provenance gaps] <proof assumptions stated? seeds logged? trace provenance pinned?>
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF/appendix>
[Artifact fixes] <additions before upload>