SIGCOMM Reproducibility
Use this before submission and again before the artifact deadline. SIGCOMM's culture treats a
result as a claim a stranger should be able to rebuild; reproducibility here is largely about
whether the network conditions behind a number are recorded well enough to recreate.
Reopen the current Call for Artifacts to confirm the process for this edition.
Evidence map
- Map each reported number to a rebuildable location: a script, a logged run, a config, and
the figure it feeds.
- For every measurement, record the topology (physical or emulated), the traffic
(workload generator or trace, with provenance), the configuration (every parameter),
and the environment (kernel, NIC, switch firmware, data-plane toolchain).
- For any stochastic result, report the run count, the seed or workload driver, and the
variance behind each percentile — a lone 99th-percentile number with no replication is not
reproducible.
- Decide early what can legally ship: production traces and topologies often cannot, so plan a
substitute with matching statistical character rather than discovering the block at the
deadline.
- Keep the paper and the artifact consistent; a figure the artifact cannot regenerate is a
reproducibility gap reviewers and the AEC both notice.
The four ledgers
| Ledger |
What it records |
Failure it prevents |
| Topology |
Nodes, links, capacities, buffer sizes, emulation vs. hardware |
"Which topology produced Figure 6?" |
| Traffic |
Workload distribution or trace, provenance, which figure used it |
Numbers that cannot be tied to an input |
| Configuration |
Every parameter and its value per experiment |
Silent knob changes between runs |
| Run |
Seeds, replication counts, timestamps, variance |
Percentiles with no reproducible basis |
Maintaining these while you run is cheap; reconstructing them after the deadline is
error-prone and often impossible.
Degrees of reproducibility
Turnkey : one command rebuilds each figure from logged runs on a documented setup
Scripted : scripts exist but need hardware, private data, or documented manual steps
Descriptive : prose detailed enough that a competent networker could rebuild the pipeline
Fragile : results depend on unrecorded conditions -> a red flag to fix before submission
For SIGCOMM, downscaled testbed results should be turnkey because evaluators actually run
them; full-scale hardware results may stay scripted with the deviation documented. State the
achieved level honestly rather than promising turnkey behavior that fails on a clean machine.
Vignette: a wide-area measurement result
A paper characterizes loss and latency on a production overlay, then proposes a routing tweak.
Its reproducibility spine: the measurement window and vantage points, the sampling method, the
anonymization applied before any release, a public-trace or synthetic substitute for the parts
that cannot ship, and a driver that regenerates the loss-latency figures from logged samples —
plus one honest sentence about which production condition the substitute cannot capture.
Legal and ethical release
- Real user or third-party-infrastructure measurement needs its handling addressed in the
paper; anonymize before release and record what was removed.
- Prefer an archival host with a DOI for anything you can ship, and document the exact reason
for anything you cannot.
Output format
[Claim inventory] <number -> rebuildable location>
[Ledger status] topology / traffic / configuration / run — complete or gaps
[Reproducibility level] turnkey / scripted / descriptive / fragile
[Data-release plan] shippable / substitute (+ provenance) / withheld (+ reason)
[Paper fixes] <must appear in the main PDF>
[Artifact fixes] <what the package still needs>
Source: brycewang-stanford/Awesome-Journal-Skills → SIGCOMM-Skills/skills/sigcomm-reproducibility/SKILL.md
1---2name: sigcomm-reproducibility3description: Use when strengthening the reproducibility evidence of an ACM SIGCOMM paper — topology and testbed ledgers, traffic workload and trace provenance, configuration and version pinning, tail-percentile run counts and variance, legal data-release decisions, and consistency between the paper's claims and the artifact that backs them.4---567# SIGCOMM Reproducibility89Use this before submission and again before the artifact deadline. SIGCOMM's culture treats a10result as a claim a stranger should be able to rebuild; reproducibility here is largely about11whether the **network conditions** behind a number are recorded well enough to recreate.12Reopen the current Call for Artifacts to confirm the process for this edition.1314## Evidence map1516- Map each reported number to a rebuildable location: a script, a logged run, a config, and17 the figure it feeds.18- For every measurement, record the **topology** (physical or emulated), the **traffic**19 (workload generator or trace, with provenance), the **configuration** (every parameter),20 and the **environment** (kernel, NIC, switch firmware, data-plane toolchain).21- For any stochastic result, report the **run count**, the seed or workload driver, and the22 variance behind each percentile — a lone 99th-percentile number with no replication is not23 reproducible.24- Decide early what can legally ship: production traces and topologies often cannot, so plan a25 substitute with matching statistical character rather than discovering the block at the26 deadline.27- Keep the paper and the artifact consistent; a figure the artifact cannot regenerate is a28 reproducibility gap reviewers and the AEC both notice.2930## The four ledgers3132| Ledger | What it records | Failure it prevents |33|---|---|---|34| Topology | Nodes, links, capacities, buffer sizes, emulation vs. hardware | "Which topology produced Figure 6?" |35| Traffic | Workload distribution or trace, provenance, which figure used it | Numbers that cannot be tied to an input |36| Configuration | Every parameter and its value per experiment | Silent knob changes between runs |37| Run | Seeds, replication counts, timestamps, variance | Percentiles with no reproducible basis |3839Maintaining these while you run is cheap; reconstructing them after the deadline is40error-prone and often impossible.4142## Degrees of reproducibility4344```text45Turnkey : one command rebuilds each figure from logged runs on a documented setup46Scripted : scripts exist but need hardware, private data, or documented manual steps47Descriptive : prose detailed enough that a competent networker could rebuild the pipeline48Fragile : results depend on unrecorded conditions -> a red flag to fix before submission49```5051For SIGCOMM, downscaled testbed results should be turnkey because evaluators actually run52them; full-scale hardware results may stay scripted with the deviation documented. State the53achieved level honestly rather than promising turnkey behavior that fails on a clean machine.5455## Vignette: a wide-area measurement result5657A paper characterizes loss and latency on a production overlay, then proposes a routing tweak.58Its reproducibility spine: the measurement window and vantage points, the sampling method, the59anonymization applied before any release, a public-trace or synthetic substitute for the parts60that cannot ship, and a driver that regenerates the loss-latency figures from logged samples —61plus one honest sentence about which production condition the substitute cannot capture.6263## Legal and ethical release6465- Real user or third-party-infrastructure measurement needs its handling addressed in the66 paper; anonymize before release and record what was removed.67- Prefer an archival host with a DOI for anything you can ship, and document the exact reason68 for anything you cannot.6970## Output format7172```text73[Claim inventory] <number -> rebuildable location>74[Ledger status] topology / traffic / configuration / run — complete or gaps75[Reproducibility level] turnkey / scripted / descriptive / fragile76[Data-release plan] shippable / substitute (+ provenance) / withheld (+ reason)77[Paper fixes] <must appear in the main PDF>78[Artifact fixes] <what the package still needs>79```8081---8283**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `SIGCOMM-Skills/skills/sigcomm-reproducibility/SKILL.md`
thedixitjain