IMC Reproducibility
Use this before submission and again before camera-ready. IMC treats availability and
reproducibility as scored dimensions, not courtesies: the submission carries an
artifact-availability declaration, accepted papers are shepherded to deliver it, and IMC runs a
dedicated Replicability Track. The goal is that a competent reader could rebuild your analysis
from the released data — and re-run your method to gather comparable new data — and reach your
conclusions.
The moving-Internet reality
Measurement differs from lab science: you cannot re-collect the same data, because the network
changes between runs. So reproducibility at IMC splits in two:
- Analysis reproducibility: the released dataset + scripts regenerate every figure and number
in the paper. This you can and must make turnkey.
- Method reproducibility: the tooling and documented vantage-point setup let someone re-run the
measurement to obtain comparable (not identical) data. This is what enables replication.
Say which you provide, and never present method reproducibility as if it reproduced your exact
numbers.
Evidence map
- Map each finding, claim, and reported number to a verifiable location — a paper section, a
figure generated from released data, or a script in the artifact.
- For the measurement, document vantage points (locations, ASes, probe types), timing
(dates, durations, cadence), targets (lists with capture dates), tools (versions,
configs), and sampling/rate limits.
- Keep the availability declaration truthful and specific: what is shared, where, and — if
something cannot be shared — exactly why (privacy, proprietary, legal).
- Keep the paper and the released data consistent: a number in the PDF that no script produces
from the released data is read as carelessness.
Availability declaration audit
| Claim in the paper |
Weak answer |
IMC-ready answer |
| "We scan N hosts" |
"Data available on request" |
Released dataset (or privacy-safe derivative) + scan metadata + scripts |
| "Measured from many vantage points" |
Vantage points unnamed |
Documented vantage-point table: locations, ASes, probe types, dates |
| "We observe behavior X over time" |
Single-snapshot data |
Time-stamped longitudinal data + the analysis window stated |
| "Our tool detects Y" |
"Code will be released" |
Released, runnable tooling with a README and a small example |
| "We used user/traffic data" |
Nothing (privacy cited vaguely) |
Aggregated/anonymized release + documented privacy method + ethics link |
Provenance pinning
[Vantage points] record every location/AS/probe-type, and quantify coverage bias
[Timing] measurement dates, durations, cadence; state the analysis window
[Targets] target/seed lists with capture dates; how they were sourced (and their stability)
[Tools] exact tool versions and configs; rate limits and opt-out/blocklist handling
[Sampling] inclusion/exclusion criteria and resulting sample sizes; how you handle churn
[Privacy] the anonymization/aggregation applied before release, matching the Ethics section
Degrees of reproducibility (state the one you achieved)
- Turnkey analysis: one documented command regenerates each figure/table from released data.
- Scripted analysis: scripts exist but need documented manual steps or restricted-data access.
- Method-only: the tooling and setup are released, but the data cannot be shared (privacy/legal)
— a stranger can re-measure, not reproduce your exact numbers.
Aim turnkey for the analysis of any releasable dataset; when data is sensitive, provide a
privacy-safe derivative plus method-only reproducibility, and say so plainly.
The Replicability Track
IMC runs a dedicated Replicability Track for work that reproduces or replicates prior
measurement results, entered via an Expression of Interest screened by a small committee, then
a full submission judged like the main track (priority to replicability over reproducibility). If
your contribution is re-measuring a prior study on today's Internet, this is the track — design
for a fair, documented re-run and an honest account of what changed and why.
Consistency and camera-ready pass
- Before submission: every scored number traces to the (anonymized) artifact; the availability
declaration matches reality; infrastructure is anonymized (no owner strings, AS/probe IDs, lab
domains).
- Before camera-ready: swap anonymized links for permanent, DOI-issuing archives; finalize the
documented schema and license; align with any Community Contribution Award intent
(
imc-artifact-evaluation).
Output format
[Claim inventory] <claim -> evidence location>
[Availability] full / partial / none, stated honestly (+ reason)
[Provenance gaps] <vantage points / timing / targets / tool versions / privacy>
[Reproducibility level] turnkey-analysis / scripted / method-only, stated honestly
[Paper fixes] <must appear in the PDF>
[Release fixes] <additions before upload / camera-ready>
1---2name: imc-reproducibility3description: Use when strengthening ACM IMC reproducibility and availability evidence, covering the artifact-availability declaration, measurement provenance (vantage points, dates, tool versions), dataset release with schema, honest reproducibility for a moving Internet, the Replicability Track, and consistency between what the paper claims and what the released data contains.4---56# IMC Reproducibility78Use this before submission and again before camera-ready. IMC treats availability and9reproducibility as scored dimensions, not courtesies: the submission carries an10**artifact-availability declaration**, accepted papers are shepherded to deliver it, and IMC runs a11dedicated **Replicability Track**. The goal is that a competent reader could rebuild your analysis12from the released data — and re-run your *method* to gather comparable new data — and reach your13conclusions.1415## The moving-Internet reality1617Measurement differs from lab science: **you cannot re-collect the same data**, because the network18changes between runs. So reproducibility at IMC splits in two:1920- **Analysis reproducibility:** the released dataset + scripts regenerate every figure and number21 in the paper. This you can and must make turnkey.22- **Method reproducibility:** the tooling and documented vantage-point setup let someone re-run the23 measurement to obtain *comparable* (not identical) data. This is what enables replication.2425Say which you provide, and never present method reproducibility as if it reproduced your exact26numbers.2728## Evidence map2930- Map each finding, claim, and reported number to a **verifiable location** — a paper section, a31 figure generated from released data, or a script in the artifact.32- For the measurement, document **vantage points** (locations, ASes, probe types), **timing**33 (dates, durations, cadence), **targets** (lists with capture dates), **tools** (versions,34 configs), and **sampling/rate limits**.35- Keep the **availability declaration** truthful and specific: what is shared, where, and — if36 something cannot be shared — exactly why (privacy, proprietary, legal).37- Keep the paper and the released data **consistent**: a number in the PDF that no script produces38 from the released data is read as carelessness.3940## Availability declaration audit4142| Claim in the paper | Weak answer | IMC-ready answer |43|---|---|---|44| "We scan N hosts" | "Data available on request" | Released dataset (or privacy-safe derivative) + scan metadata + scripts |45| "Measured from many vantage points" | Vantage points unnamed | Documented vantage-point table: locations, ASes, probe types, dates |46| "We observe behavior X over time" | Single-snapshot data | Time-stamped longitudinal data + the analysis window stated |47| "Our tool detects Y" | "Code will be released" | Released, runnable tooling with a README and a small example |48| "We used user/traffic data" | Nothing (privacy cited vaguely) | Aggregated/anonymized release + documented privacy method + ethics link |4950## Provenance pinning5152```text53[Vantage points] record every location/AS/probe-type, and quantify coverage bias54[Timing] measurement dates, durations, cadence; state the analysis window55[Targets] target/seed lists with capture dates; how they were sourced (and their stability)56[Tools] exact tool versions and configs; rate limits and opt-out/blocklist handling57[Sampling] inclusion/exclusion criteria and resulting sample sizes; how you handle churn58[Privacy] the anonymization/aggregation applied before release, matching the Ethics section59```6061## Degrees of reproducibility (state the one you achieved)6263- **Turnkey analysis:** one documented command regenerates each figure/table from released data.64- **Scripted analysis:** scripts exist but need documented manual steps or restricted-data access.65- **Method-only:** the tooling and setup are released, but the data cannot be shared (privacy/legal)66 — a stranger can re-measure, not reproduce your exact numbers.6768Aim turnkey for the analysis of any releasable dataset; when data is sensitive, provide a69privacy-safe derivative plus method-only reproducibility, and say so plainly.7071## The Replicability Track7273IMC runs a dedicated **Replicability Track** for work that reproduces or replicates prior74measurement results, entered via an **Expression of Interest** screened by a small committee, then75a full submission judged like the main track (priority to replicability over reproducibility). If76your contribution *is* re-measuring a prior study on today's Internet, this is the track — design77for a fair, documented re-run and an honest account of what changed and why.7879## Consistency and camera-ready pass8081- Before submission: every scored number traces to the (anonymized) artifact; the availability82 declaration matches reality; infrastructure is anonymized (no owner strings, AS/probe IDs, lab83 domains).84- Before camera-ready: swap anonymized links for permanent, DOI-issuing archives; finalize the85 documented schema and license; align with any Community Contribution Award intent86 (`imc-artifact-evaluation`).8788## Output format8990```text91[Claim inventory] <claim -> evidence location>92[Availability] full / partial / none, stated honestly (+ reason)93[Provenance gaps] <vantage points / timing / targets / tool versions / privacy>94[Reproducibility level] turnkey-analysis / scripted / method-only, stated honestly95[Paper fixes] <must appear in the PDF>96[Release fixes] <additions before upload / camera-ready>97```
brycewang-stanford