Socc Reproducibility

SoCC Reproducibility

Use this before submission and again before camera-ready. SoCC — the joint SIGMOD+SIGOPS cloud symposium — is read by reviewers who expect an inspectable measurement trail: the SIGOPS half wants to believe the system runs, and the SIGMOD half wants to believe the numbers. The goal is that a competent reader with a comparable testbed could rebuild your evidence and reach your conclusions — including the tail latency and cost, not just the average.

Evidence map

• Map each cloud claim and reported number to a verifiable location — a paper section, a figure generated from logged runs, or a script in the artifact.
• For systems, give enough of the mechanism, configuration, and testbed (node counts, instance types, OS/kernel, network) that a reader could re-deploy and re-measure.
• For measurement/trace studies, report the trace provenance, extraction date, filtering, the workload replay, metrics, and the analysis scripts.
• Reproduce tail and cost: percentiles (p95/p99/p99.9), the pricing model behind any cost claim, and the number of runs and variance — the mean alone is not a cloud result.
• Keep the paper and the artifact consistent: a number in the PDF that no script regenerates is the contradiction reviewers read as carelessness.

Reproducibility statement audit

"Available on request" is treated as not available; convert every such line into a concrete, anonymized (then released) artifact or an explicit, justified exception (e.g., a confidential production trace, with a synthetic generator provided instead).

Provenance pinning

[Measurement] pin commit SHAs; record trace extraction dates; archive the replayed trace or a
              faithful generator, not just a query or a pointer
[Testbed]     record node counts, instance types, OS/kernel versions, network, and the run count;
              a cloud result that cannot be re-deployed cannot be reproduced
[Cost]        state the pricing model and the source of every cost figure so a reader can recompute
[Tail]        log per-request or per-run latency distributions, not only aggregates
[Randomness]  log seeds for stochastic components; say what is and is not deterministic

Degrees of reproducibility (state the one you achieved)

• Turnkey: one documented command regenerates each figure (including tail and cost) from logged runs, at least at reduced scale.
• Scripted: scripts exist but require a specific testbed, documented manual steps, or restricted trace access.
• Descriptive: prose and configuration detailed enough that a competent reader could rebuild the deployment and measurement pipeline.

For SoCC, aim turnkey for anything an evaluator could rerun at small scale (a short trace replay, a tail/cost plot from logged runs); full-cluster or proprietary-trace results may stay scripted with access clearly documented. Stating the achieved level honestly beats promising turnkey behavior that fails on someone else's testbed.

Vignette: a scheduling measurement paper

Consider a paper measuring a new scheduler on a replayed production trace. Its reproducibility spine: the scheduler code with pinned SHAs; the replay harness and the (anonymized, then released) trace with its extraction date; the testbed description (nodes, instance types, kernel); the measurement scripts that turn raw logs into the throughput, p99, and cost figures; the run count and variance; and one honest sentence about the parts (a confidential production trace, the full cluster) that cannot be shared and what synthetic or scaled substitute is provided.

Consistency and camera-ready pass

• Before submission: every reported number traces to the artifact; the reproducibility statement matches reality; the artifact is anonymized (no cluster names, provider hints, or trace provenance that reveals identity).
• Before camera-ready: swap anonymized links for permanent, DOI-issuing archives, and align the statement with the ACM badges you are pursuing if the edition offers evaluation (socc-artifact-evaluation).

Output format

[Claim inventory] <claim -> evidence location>
[Reproducibility statement] concrete / vague / missing
[Provenance gaps] <trace SHAs+dates / testbed description / cost model / tail logging / seeds>
[Tail + cost] reproducible, not just the mean? yes/no
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF>
[Artifact fixes] <additions before upload>

Source: brycewang-stanford/Awesome-Journal-Skills → SoCC-Skills/skills/socc-reproducibility/SKILL.md