NDSS Experiments
Evidence at NDSS answers one question: would this hold on the real network? Every design
choice in the evaluation — target selection, vantage, scale, baseline — either supports
that answer or advertises that you avoided it.
Evidence by contribution type
| You claim... |
Reviewers require... |
Weak substitute they will flag |
| A new attack |
End-to-end demonstration on real implementations, named versions, measured preconditions and cost |
PoC against a config you weakened |
| A prevalent condition |
Defined population, sampling method, validated vulnerability test, false-positive analysis |
Extrapolation from anecdotes |
| A defense |
Security and utility: adaptive-adversary evaluation + overhead on realistic workloads |
Blocking only the published attack |
| A detection system |
Base-rate honest metrics on traffic resembling deployment, drift discussion |
Accuracy on one stale benchmark |
| A protocol change |
Interop and incremental-deployment story, behavior under partial adoption |
"Assume everyone upgrades" |
Choosing the experimental substrate
Match the substrate to the claim, and say why in the paper:
- Live Internet — the only substrate for prevalence claims. Requires the ethics
machinery (below) and a recorded snapshot: measurement results are time-indexed facts, so
log the observation window, vantage points, and population source.
- Testbed / physical devices — for exploit reliability and defense overhead, where you
need ground truth and repeatability. Pin firmware, kernel, and topology; a testbed nobody
can reconstruct is an anecdote with racks.
- Emulation / simulation — legitimate for scale sweeps and what-if topologies after
the mechanism is shown real elsewhere. A paper whose only evidence is simulated packets
is, at this venue, a proposal.
Mixed designs are the norm: demonstrate on real systems, generalize on the testbed, sweep
in emulation, and keep the chain of custody between the three explicit.
The adaptive-adversary requirement
For defenses, the reviewer's first move is to play attacker against your design. Do it
first, in print:
- Restate the defense as the constraint it imposes on the adversary.
- Enumerate strategies that respect capabilities from the threat model but target the
mechanism itself — evasion, mimicry, resource exhaustion, downgrade, oracle abuse.
- Implement the strongest of these, not the most convenient, and report where the defense
bends. Cost asymmetry (defender cents vs. attacker weeks) is a legitimate result;
silence about adaptation is not.
Measurement hygiene
- One vantage point is a case study; state how many you used and what each can and cannot
see (anycast, geo-blocking, and CDN behavior all bite here).
- Validate the classifier or vulnerability test against labeled ground truth and report
both error directions — a scanner's false positives are the headline number's error bar.
- Repeat over time before claiming stability; the Internet's diurnal and patch-cycle
rhythms confound single-shot scans.
Ethics as an experimental parameter
Live experiments are bounded by harm, and the CFP backs this with an Ethics Review Board.
Design limits in from the start: rate caps and opt-out honoring for scans, no serving of
real user traffic during interception experiments, synthetic victims wherever the
demonstration allows, notification pipelines for anything you confirm exploitable, and IRB
engagement where humans are observed — while remembering NDSS treats IRB sign-off as
necessary context, not sufficient mitigation. Every one of these choices belongs in the
paper's method text, not only in the ethics section.
Reporting discipline
For every headline number, the text or appendix must pin:
target set + versions + selection method
vantage(s) and observation window
trials, variance, and the aggregation rule
hardware/software of the measurement or attack host
configuration deltas from defaults (ideally: none, or justified)
the command or procedure that regenerates the number (→ artifact)
Numbers that move between runs get intervals; comparisons against baselines get identical
conditions or an explanation of why identity was impossible.
Failure modes this venue punishes
- Evaluating on the environment you developed on, and nowhere else.
- Baselines run with defaults while your system got tuning.
- Success rates without denominators; "up to X" as a summary statistic.
- Prevalence claims from a population chosen because it was scannable, described as if it
were representative.
Output format
[Claim → evidence map] each headline claim + substrate + status
[Adaptive evaluation] strategies enumerated / strongest implemented / result
[Measurement validity] population, vantage, validation, error directions
[Ethics parameters] rate caps, synthetic victims, notification, IRB state
[Repro pinning] items from the reporting block still missing
1---2name: ndss-experiments3description: Use when designing or auditing the evaluation of an NDSS paper — attack demonstrations, defense evaluations under adaptive adversaries, and Internet measurements — choosing between testbed, emulation, and live experiments and meeting NDSS's deployment-realism evidence bar.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/ndss-experiments6---78# NDSS Experiments910Evidence at NDSS answers one question: *would this hold on the real network?* Every design11choice in the evaluation — target selection, vantage, scale, baseline — either supports12that answer or advertises that you avoided it.1314## Evidence by contribution type1516| You claim... | Reviewers require... | Weak substitute they will flag |17| --- | --- | --- |18| A new attack | End-to-end demonstration on real implementations, named versions, measured preconditions and cost | PoC against a config you weakened |19| A prevalent condition | Defined population, sampling method, validated vulnerability test, false-positive analysis | Extrapolation from anecdotes |20| A defense | Security *and* utility: adaptive-adversary evaluation + overhead on realistic workloads | Blocking only the published attack |21| A detection system | Base-rate honest metrics on traffic resembling deployment, drift discussion | Accuracy on one stale benchmark |22| A protocol change | Interop and incremental-deployment story, behavior under partial adoption | "Assume everyone upgrades" |2324## Choosing the experimental substrate2526Match the substrate to the claim, and say why in the paper:2728- **Live Internet** — the only substrate for prevalence claims. Requires the ethics29 machinery (below) and a recorded snapshot: measurement results are time-indexed facts, so30 log the observation window, vantage points, and population source.31- **Testbed / physical devices** — for exploit reliability and defense overhead, where you32 need ground truth and repeatability. Pin firmware, kernel, and topology; a testbed nobody33 can reconstruct is an anecdote with racks.34- **Emulation / simulation** — legitimate for scale sweeps and what-if topologies *after*35 the mechanism is shown real elsewhere. A paper whose only evidence is simulated packets36 is, at this venue, a proposal.3738Mixed designs are the norm: demonstrate on real systems, generalize on the testbed, sweep39in emulation, and keep the chain of custody between the three explicit.4041## The adaptive-adversary requirement4243For defenses, the reviewer's first move is to play attacker against your design. Do it44first, in print:45461. Restate the defense as the constraint it imposes on the adversary.472. Enumerate strategies that respect capabilities from the threat model but target the48 mechanism itself — evasion, mimicry, resource exhaustion, downgrade, oracle abuse.493. Implement the strongest of these, not the most convenient, and report where the defense50 bends. Cost asymmetry (defender cents vs. attacker weeks) is a legitimate result;51 silence about adaptation is not.5253## Measurement hygiene5455- One vantage point is a case study; state how many you used and what each can and cannot56 see (anycast, geo-blocking, and CDN behavior all bite here).57- Validate the classifier or vulnerability test against labeled ground truth and report58 both error directions — a scanner's false positives *are* the headline number's error bar.59- Repeat over time before claiming stability; the Internet's diurnal and patch-cycle60 rhythms confound single-shot scans.6162## Ethics as an experimental parameter6364Live experiments are bounded by harm, and the CFP backs this with an Ethics Review Board.65Design limits in from the start: rate caps and opt-out honoring for scans, no serving of66real user traffic during interception experiments, synthetic victims wherever the67demonstration allows, notification pipelines for anything you confirm exploitable, and IRB68engagement where humans are observed — while remembering NDSS treats IRB sign-off as69necessary context, not sufficient mitigation. Every one of these choices belongs in the70paper's method text, not only in the ethics section.7172## Reporting discipline7374```text75For every headline number, the text or appendix must pin:76 target set + versions + selection method77 vantage(s) and observation window78 trials, variance, and the aggregation rule79 hardware/software of the measurement or attack host80 configuration deltas from defaults (ideally: none, or justified)81 the command or procedure that regenerates the number (→ artifact)82```8384Numbers that move between runs get intervals; comparisons against baselines get identical85conditions or an explanation of why identity was impossible.8687## Failure modes this venue punishes8889- Evaluating on the environment you developed on, and nowhere else.90- Baselines run with defaults while your system got tuning.91- Success rates without denominators; "up to X" as a summary statistic.92- Prevalence claims from a population chosen because it was scannable, described as if it93 were representative.9495## Output format9697```text98[Claim → evidence map] each headline claim + substrate + status99[Adaptive evaluation] strategies enumerated / strongest implemented / result100[Measurement validity] population, vantage, validation, error directions101[Ethics parameters] rate caps, synthetic victims, notification, IRB state102[Repro pinning] items from the reporting block still missing103```