IMC Reproducibility
Use this before submission and again before camera-ready. IMC treats availability and
reproducibility as scored dimensions, not courtesies: the submission carries an
artifact-availability declaration, accepted papers are shepherded to deliver it, and IMC runs a
dedicated Replicability Track. The goal is that a competent reader could rebuild your analysis
from the released data — and re-run your method to gather comparable new data — and reach your
conclusions.
The moving-Internet reality
Measurement differs from lab science: you cannot re-collect the same data, because the network
changes between runs. So reproducibility at IMC splits in two:
- Analysis reproducibility: the released dataset + scripts regenerate every figure and number
in the paper. This you can and must make turnkey.
- Method reproducibility: the tooling and documented vantage-point setup let someone re-run the
measurement to obtain comparable (not identical) data. This is what enables replication.
Say which you provide, and never present method reproducibility as if it reproduced your exact
numbers.
Evidence map
- Map each finding, claim, and reported number to a verifiable location — a paper section, a
figure generated from released data, or a script in the artifact.
- For the measurement, document vantage points (locations, ASes, probe types), timing
(dates, durations, cadence), targets (lists with capture dates), tools (versions,
configs), and sampling/rate limits.
- Keep the availability declaration truthful and specific: what is shared, where, and — if
something cannot be shared — exactly why (privacy, proprietary, legal).
- Keep the paper and the released data consistent: a number in the PDF that no script produces
from the released data is read as carelessness.
Availability declaration audit
| Claim in the paper |
Weak answer |
IMC-ready answer |
| "We scan N hosts" |
"Data available on request" |
Released dataset (or privacy-safe derivative) + scan metadata + scripts |
| "Measured from many vantage points" |
Vantage points unnamed |
Documented vantage-point table: locations, ASes, probe types, dates |
| "We observe behavior X over time" |
Single-snapshot data |
Time-stamped longitudinal data + the analysis window stated |
| "Our tool detects Y" |
"Code will be released" |
Released, runnable tooling with a README and a small example |
| "We used user/traffic data" |
Nothing (privacy cited vaguely) |
Aggregated/anonymized release + documented privacy method + ethics link |
Provenance pinning
[Vantage points] record every location/AS/probe-type, and quantify coverage bias
[Timing] measurement dates, durations, cadence; state the analysis window
[Targets] target/seed lists with capture dates; how they were sourced (and their stability)
[Tools] exact tool versions and configs; rate limits and opt-out/blocklist handling
[Sampling] inclusion/exclusion criteria and resulting sample sizes; how you handle churn
[Privacy] the anonymization/aggregation applied before release, matching the Ethics section
Degrees of reproducibility (state the one you achieved)
- Turnkey analysis: one documented command regenerates each figure/table from released data.
- Scripted analysis: scripts exist but need documented manual steps or restricted-data access.
- Method-only: the tooling and setup are released, but the data cannot be shared (privacy/legal)
— a stranger can re-measure, not reproduce your exact numbers.
Aim turnkey for the analysis of any releasable dataset; when data is sensitive, provide a
privacy-safe derivative plus method-only reproducibility, and say so plainly.
The Replicability Track
IMC runs a dedicated Replicability Track for work that reproduces or replicates prior
measurement results, entered via an Expression of Interest screened by a small committee, then
a full submission judged like the main track (priority to replicability over reproducibility). If
your contribution is re-measuring a prior study on today's Internet, this is the track — design
for a fair, documented re-run and an honest account of what changed and why.
Consistency and camera-ready pass
- Before submission: every scored number traces to the (anonymized) artifact; the availability
declaration matches reality; infrastructure is anonymized (no owner strings, AS/probe IDs, lab
domains).
- Before camera-ready: swap anonymized links for permanent, DOI-issuing archives; finalize the
documented schema and license; align with any Community Contribution Award intent
(
imc-artifact-evaluation).
Output format
[Claim inventory] <claim -> evidence location>
[Availability] full / partial / none, stated honestly (+ reason)
[Provenance gaps] <vantage points / timing / targets / tool versions / privacy>
[Reproducibility level] turnkey-analysis / scripted / method-only, stated honestly
[Paper fixes] <must appear in the PDF>
[Release fixes] <additions before upload / camera-ready>
Source: brycewang-stanford/Awesome-Journal-Skills → IMC-Skills/skills/imc-reproducibility/SKILL.md
1---2name: imc-reproducibility3description: Use when strengthening ACM IMC reproducibility and availability evidence, covering the artifact-availability declaration, measurement provenance (vantage points, dates, tool versions), dataset release with schema, honest reproducibility for a moving Internet, the Replicability Track, and consistency between what the paper claims and what the released data contains.4---567# IMC Reproducibility89Use this before submission and again before camera-ready. IMC treats availability and10reproducibility as scored dimensions, not courtesies: the submission carries an11**artifact-availability declaration**, accepted papers are shepherded to deliver it, and IMC runs a12dedicated **Replicability Track**. The goal is that a competent reader could rebuild your analysis13from the released data — and re-run your *method* to gather comparable new data — and reach your14conclusions.1516## The moving-Internet reality1718Measurement differs from lab science: **you cannot re-collect the same data**, because the network19changes between runs. So reproducibility at IMC splits in two:2021- **Analysis reproducibility:** the released dataset + scripts regenerate every figure and number22 in the paper. This you can and must make turnkey.23- **Method reproducibility:** the tooling and documented vantage-point setup let someone re-run the24 measurement to obtain *comparable* (not identical) data. This is what enables replication.2526Say which you provide, and never present method reproducibility as if it reproduced your exact27numbers.2829## Evidence map3031- Map each finding, claim, and reported number to a **verifiable location** — a paper section, a32 figure generated from released data, or a script in the artifact.33- For the measurement, document **vantage points** (locations, ASes, probe types), **timing**34 (dates, durations, cadence), **targets** (lists with capture dates), **tools** (versions,35 configs), and **sampling/rate limits**.36- Keep the **availability declaration** truthful and specific: what is shared, where, and — if37 something cannot be shared — exactly why (privacy, proprietary, legal).38- Keep the paper and the released data **consistent**: a number in the PDF that no script produces39 from the released data is read as carelessness.4041## Availability declaration audit4243| Claim in the paper | Weak answer | IMC-ready answer |44|---|---|---|45| "We scan N hosts" | "Data available on request" | Released dataset (or privacy-safe derivative) + scan metadata + scripts |46| "Measured from many vantage points" | Vantage points unnamed | Documented vantage-point table: locations, ASes, probe types, dates |47| "We observe behavior X over time" | Single-snapshot data | Time-stamped longitudinal data + the analysis window stated |48| "Our tool detects Y" | "Code will be released" | Released, runnable tooling with a README and a small example |49| "We used user/traffic data" | Nothing (privacy cited vaguely) | Aggregated/anonymized release + documented privacy method + ethics link |5051## Provenance pinning5253```text54[Vantage points] record every location/AS/probe-type, and quantify coverage bias55[Timing] measurement dates, durations, cadence; state the analysis window56[Targets] target/seed lists with capture dates; how they were sourced (and their stability)57[Tools] exact tool versions and configs; rate limits and opt-out/blocklist handling58[Sampling] inclusion/exclusion criteria and resulting sample sizes; how you handle churn59[Privacy] the anonymization/aggregation applied before release, matching the Ethics section60```6162## Degrees of reproducibility (state the one you achieved)6364- **Turnkey analysis:** one documented command regenerates each figure/table from released data.65- **Scripted analysis:** scripts exist but need documented manual steps or restricted-data access.66- **Method-only:** the tooling and setup are released, but the data cannot be shared (privacy/legal)67 — a stranger can re-measure, not reproduce your exact numbers.6869Aim turnkey for the analysis of any releasable dataset; when data is sensitive, provide a70privacy-safe derivative plus method-only reproducibility, and say so plainly.7172## The Replicability Track7374IMC runs a dedicated **Replicability Track** for work that reproduces or replicates prior75measurement results, entered via an **Expression of Interest** screened by a small committee, then76a full submission judged like the main track (priority to replicability over reproducibility). If77your contribution *is* re-measuring a prior study on today's Internet, this is the track — design78for a fair, documented re-run and an honest account of what changed and why.7980## Consistency and camera-ready pass8182- Before submission: every scored number traces to the (anonymized) artifact; the availability83 declaration matches reality; infrastructure is anonymized (no owner strings, AS/probe IDs, lab84 domains).85- Before camera-ready: swap anonymized links for permanent, DOI-issuing archives; finalize the86 documented schema and license; align with any Community Contribution Award intent87 (`imc-artifact-evaluation`).8889## Output format9091```text92[Claim inventory] <claim -> evidence location>93[Availability] full / partial / none, stated honestly (+ reason)94[Provenance gaps] <vantage points / timing / targets / tool versions / privacy>95[Reproducibility level] turnkey-analysis / scripted / method-only, stated honestly96[Paper fixes] <must appear in the PDF>97[Release fixes] <additions before upload / camera-ready>98```99100---101102**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `IMC-Skills/skills/imc-reproducibility/SKILL.md`
thedixitjain