thedixitjain NDSS Experiments
Evidence at NDSS answers one question: would this hold on the real network? Every design choice in the evaluation — target selection, vantage, scale, baseline — either supports that answer or advertises that you avoided it.
Evidence by contribution type
| You claim... | Reviewers require... | Weak substitute they will flag |
|---|---|---|
| A new attack | End-to-end demonstration on real implementations, named versions, measured preconditions and cost | PoC against a config you weakened |
| A prevalent condition | Defined population, sampling method, validated vulnerability test, false-positive analysis | Extrapolation from anecdotes |
| A defense | Security and utility: adaptive-adversary evaluation + overhead on realistic workloads | Blocking only the published attack |
| A detection system | Base-rate honest metrics on traffic resembling deployment, drift discussion | Accuracy on one stale benchmark |
| A protocol change | Interop and incremental-deployment story, behavior under partial adoption | "Assume everyone upgrades" |
Choosing the experimental substrate
Match the substrate to the claim, and say why in the paper:
- Live Internet — the only substrate for prevalence claims. Requires the ethics machinery (below) and a recorded snapshot: measurement results are time-indexed facts, so log the observation window, vantage points, and population source.
- Testbed / physical devices — for exploit reliability and defense overhead, where you need ground truth and repeatability. Pin firmware, kernel, and topology; a testbed nobody can reconstruct is an anecdote with racks.
- Emulation / simulation — legitimate for scale sweeps and what-if topologies after the mechanism is shown real elsewhere. A paper whose only evidence is simulated packets is, at this venue, a proposal.
Mixed designs are the norm: demonstrate on real systems, generalize on the testbed, sweep in emulation, and keep the chain of custody between the three explicit.
The adaptive-adversary requirement
For defenses, the reviewer's first move is to play attacker against your design. Do it first, in print:
- Restate the defense as the constraint it imposes on the adversary.
- Enumerate strategies that respect capabilities from the threat model but target the mechanism itself — evasion, mimicry, resource exhaustion, downgrade, oracle abuse.
- Implement the strongest of these, not the most convenient, and report where the defense bends. Cost asymmetry (defender cents vs. attacker weeks) is a legitimate result; silence about adaptation is not.
Measurement hygiene
- One vantage point is a case study; state how many you used and what each can and cannot see (anycast, geo-blocking, and CDN behavior all bite here).
- Validate the classifier or vulnerability test against labeled ground truth and report both error directions — a scanner's false positives are the headline number's error bar.
- Repeat over time before claiming stability; the Internet's diurnal and patch-cycle rhythms confound single-shot scans.
Ethics as an experimental parameter
Live experiments are bounded by harm, and the CFP backs this with an Ethics Review Board. Design limits in from the start: rate caps and opt-out honoring for scans, no serving of real user traffic during interception experiments, synthetic victims wherever the demonstration allows, notification pipelines for anything you confirm exploitable, and IRB engagement where humans are observed — while remembering NDSS treats IRB sign-off as necessary context, not sufficient mitigation. Every one of these choices belongs in the paper's method text, not only in the ethics section.
Reporting discipline
For every headline number, the text or appendix must pin:
target set + versions + selection method
vantage(s) and observation window
trials, variance, and the aggregation rule
hardware/software of the measurement or attack host
configuration deltas from defaults (ideally: none, or justified)
the command or procedure that regenerates the number (→ artifact)
Numbers that move between runs get intervals; comparisons against baselines get identical conditions or an explanation of why identity was impossible.
Failure modes this venue punishes
- Evaluating on the environment you developed on, and nowhere else.
- Baselines run with defaults while your system got tuning.
- Success rates without denominators; "up to X" as a summary statistic.
- Prevalence claims from a population chosen because it was scannable, described as if it were representative.
Output format
[Claim → evidence map] each headline claim + substrate + status
[Adaptive evaluation] strategies enumerated / strongest implemented / result
[Measurement validity] population, vantage, validation, error directions
[Ethics parameters] rate caps, synthetic victims, notification, IRB state
[Repro pinning] items from the reporting block still missing
Source: brycewang-stanford/Awesome-Journal-Skills → NDSS-Skills/skills/ndss-experiments/SKILL.md