IMC Measurements
Use this before submission when the measurement design is not yet locked. IMC reviewers are
measurement empiricists; the study is where a good question is won or lost. The organizing
principle is evidence proportional to a claim about the real Internet — the measurement must
observe the thing the paper asserts, from vantage points and over a period a skeptic would accept,
and it must be collected safely and ethically.
Design audit
- Match the measurement to the claim shape. A claim about reachability in the wild needs many
real vantage points; a claim about deployment needs coverage of the population; a claim about
behavior over time needs a dated longitudinal window; a claim about security needs validated
ground truth. A snapshot from two hosts cannot support a wild-Internet claim.
- Choose vantage points for representativeness, not convenience. Name locations, ASes, and
probe types; argue why they support the claim; and quantify the coverage bias you cannot
remove (e.g., volunteer probes over-representing certain regions).
- Get ground truth right. For detection/labeling claims, state where the labels come from and
validate a subsample against an independent source; report the residual error.
- Design for a moving Internet. Instrument for churn, diurnal effects, load-balancing, CDN and
anycast behavior, and path changes. Decide before collecting what a change over time would mean.
- Make active measurement safe and ethical (see the safety block) — this is inseparable from
the design at IMC, not a compliance afterthought.
- Pin provenance so the measurement can be re-run as a method and its data reproduced as an
analysis (
imc-reproducibility).
Claim-to-evidence design table
| Claim about the Internet |
Matching evidence |
Reject pattern avoided |
| "Reachable/blocked in the wild" |
Many real vantage points across networks/regions, dated |
"Two hosts stood in for the Internet" |
| "Widely deployed" |
Population-scale coverage with stated sampling |
"Convenience sample, claimed universal" |
| "Behavior changes over time" |
Dated longitudinal window with stability analysis |
"Single snapshot presented as a trend" |
| "We detect/measure X accurately" |
Validated ground-truth subsample + error bounds |
"No ground truth; labels unvalidated" |
| "This vantage point is representative" |
Coverage-bias quantified vs. the target population |
"Bias assumed away" |
Ethical and safe active measurement
At IMC this is design, and it is a review gate (imc-submission):
[Do no harm] rate-limit probes; avoid overloading targets or intermediary networks
[Opt-out] honor blocklists and abuse contacts; provide a clear opt-out and project page
[Human data] for traffic/DNS/user data, obtain IRB approval/exemption; minimize and anonymize
[Belmont] argue respect for persons (consent where owed), beneficence (risk vs. benefit),
justice (who bears risk vs. benefits)
[Disclosure] for vulnerability/exposure findings, plan responsible disclosure and a timeline
[Consent] for volunteer vantage points, obtain informed consent and protect participants
Provenance floor for measurement studies
- Record every vantage point (location, AS, probe type) and quantify coverage.
- Record timing: measurement dates, durations, cadence, and the analysis window.
- Record targets: seed/target lists with capture dates and how they were sourced.
- Record tools: exact versions, configs, rate limits, and opt-out handling.
- Archive the captured data, not just the query — the Internet will have changed by re-run.
Vignette: measuring protocol deployment
Suppose the paper claims a protocol is widely deployed. The matching plan: scan (or query a
platform for) a population-scale target set with stated inclusion criteria; run from vantage points
whose coverage you quantify; date every scan and repeat over a window to show stability; validate a
subsample of "deployed" classifications against an independent signal; handle churn and duplicates
explicitly; and document the probing safety design in the Ethics section — every number traceable
to a dated, provenanced capture in the released dataset.
Reporting floor
- Confidence intervals for rate/proportion estimates; say what they represent and the sample size.
- Coverage and its bias, stated quantitatively, for every vantage-point-based claim.
- Measurement dates and windows for every temporal claim.
- The compute/probe budget and any rate limits actually used.
Output format
[Measurement readiness] strong / adequate / weak
[Claim -> evidence map] <claim: vantage points / window / ground truth / statistic>
[Representativeness] <vantage points named? coverage bias quantified?>
[Ethics/safety] <active-measurement safety + IRB + disclosure handled? yes/no>
[Provenance] <vantage points / timing / targets / tool versions pinned? yes/no>
[Decision-critical next run] <one measurement to add or extend>
Source: brycewang-stanford/Awesome-Journal-Skills → IMC-Skills/skills/imc-experiments/SKILL.md
1---2name: imc-experiments3description: Use when designing or auditing ACM IMC measurements, covering representative vantage points, honest ground truth, longitudinal design for a moving Internet, safe and ethical active measurement, coverage-bias quantification, provenance pinning, and matching the measurement to the shape of each claim about the real Internet.4---5
6
7# IMC Measurements
8
9Use this before submission when the measurement design is not yet locked. IMC reviewers are
10measurement empiricists; the study is where a good question is won or lost. The organizing
11principle is **evidence proportional to a claim about the real Internet** — the measurement must
12observe the thing the paper asserts, from vantage points and over a period a skeptic would accept,
13and it must be collected safely and ethically.
14
15## Design audit
16
17- **Match the measurement to the claim shape.** A claim about *reachability in the wild* needs many
18 real vantage points; a claim about *deployment* needs coverage of the population; a claim about
19 *behavior over time* needs a dated longitudinal window; a claim about *security* needs validated
20 ground truth. A snapshot from two hosts cannot support a wild-Internet claim.
21- **Choose vantage points for representativeness, not convenience.** Name locations, ASes, and
22 probe types; argue why they support the claim; and **quantify the coverage bias** you cannot
23 remove (e.g., volunteer probes over-representing certain regions).
24- **Get ground truth right.** For detection/labeling claims, state where the labels come from and
25 validate a subsample against an independent source; report the residual error.
26- **Design for a moving Internet.** Instrument for churn, diurnal effects, load-balancing, CDN and
27 anycast behavior, and path changes. Decide before collecting what a change over time would mean.
28- **Make active measurement safe and ethical** (see the safety block) — this is inseparable from
29 the design at IMC, not a compliance afterthought.
30- **Pin provenance** so the measurement can be re-run as a *method* and its data reproduced as an
31 *analysis* (`imc-reproducibility`).
32
33## Claim-to-evidence design table
34
35| Claim about the Internet | Matching evidence | Reject pattern avoided |
36|---|---|---|
37| "Reachable/blocked in the wild" | Many real vantage points across networks/regions, dated | "Two hosts stood in for the Internet" |
38| "Widely deployed" | Population-scale coverage with stated sampling | "Convenience sample, claimed universal" |
39| "Behavior changes over time" | Dated longitudinal window with stability analysis | "Single snapshot presented as a trend" |
40| "We detect/measure X accurately" | Validated ground-truth subsample + error bounds | "No ground truth; labels unvalidated" |
41| "This vantage point is representative" | Coverage-bias quantified vs. the target population | "Bias assumed away" |
42
43## Ethical and safe active measurement
44
45At IMC this is design, and it is a review gate (`imc-submission`):
46
47```text
48[Do no harm] rate-limit probes; avoid overloading targets or intermediary networks
49[Opt-out] honor blocklists and abuse contacts; provide a clear opt-out and project page
50[Human data] for traffic/DNS/user data, obtain IRB approval/exemption; minimize and anonymize
51[Belmont] argue respect for persons (consent where owed), beneficence (risk vs. benefit),
52 justice (who bears risk vs. benefits)
53[Disclosure] for vulnerability/exposure findings, plan responsible disclosure and a timeline
54[Consent] for volunteer vantage points, obtain informed consent and protect participants
55```
56
57## Provenance floor for measurement studies
58
59- Record every **vantage point** (location, AS, probe type) and quantify coverage.
60- Record **timing**: measurement dates, durations, cadence, and the analysis window.
61- Record **targets**: seed/target lists with capture dates and how they were sourced.
62- Record **tools**: exact versions, configs, rate limits, and opt-out handling.
63- Archive the **captured data**, not just the query — the Internet will have changed by re-run.
64
65## Vignette: measuring protocol deployment
66
67Suppose the paper claims a protocol is widely deployed. The matching plan: scan (or query a
68platform for) a population-scale target set with stated inclusion criteria; run from vantage points
69whose coverage you quantify; date every scan and repeat over a window to show stability; validate a
70subsample of "deployed" classifications against an independent signal; handle churn and duplicates
71explicitly; and document the probing safety design in the Ethics section — every number traceable
72to a dated, provenanced capture in the released dataset.
73
74## Reporting floor
75
76- Confidence intervals for rate/proportion estimates; say what they represent and the sample size.
77- Coverage and its bias, stated quantitatively, for every vantage-point-based claim.
78- Measurement dates and windows for every temporal claim.
79- The compute/probe budget and any rate limits actually used.
80
81## Output format
82
83```text
84[Measurement readiness] strong / adequate / weak
85[Claim -> evidence map] <claim: vantage points / window / ground truth / statistic>
86[Representativeness] <vantage points named? coverage bias quantified?>
87[Ethics/safety] <active-measurement safety + IRB + disclosure handled? yes/no>
88[Provenance] <vantage points / timing / targets / tool versions pinned? yes/no>
89[Decision-critical next run] <one measurement to add or extend>
90```
91
92---
93
94**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `IMC-Skills/skills/imc-experiments/SKILL.md`
thedixitjain