brycewang-stanford NDSS Artifact Evaluation
NDSS added artifact evaluation to the symposium in recent editions (running on the
community secartifacts platform, verified for 2024-2026), and it is opt-in after
conditional acceptance — a separate track, not a submission requirement. Papers that pass
carry badges on the first page and add a 2-page artifact appendix (Call for Artifacts,
checked 2026-07-08). Decide early whether you are engineering for it, because the packaging
choices are cheap during the project and expensive afterward.
The three badges
| Badge | Claim to the committee | What you must ship |
|---|---|---|
| Available | The artifact is publicly, permanently retrievable | Deposit in an archival repository with a DOI (Zenodo, figshare); a bare GitHub URL is not "available" |
| Functional | It does what the paper says, is usable and documented | Build/run instructions that work on a clean machine; the components matching the paper's claims |
| Reproduced | The paper's results can be regenerated by others | One documented path per headline result, with expected output and tolerances |
Badges are independent — you can earn Available and Functional without Reproduced when the world (live targets, proprietary data) genuinely prevents re-execution. Claim only what you can defend; an evaluator who cannot reproduce a "Reproduced" claim writes that in the report.
From claims ledger to artifact
The claims.yml ledger built in ndss-reproducibility is the artifact's spine. Convert it
into an evaluator-facing package:
artifact/
README.md # what this is, badges sought, hardware/time needs, safety notes
INSTALL.md # clean-machine setup; pinned deps / container image
claims/ # one entry per headline result:
fig4/ run.sh # → expected_output/, runtime + resource note
tab2/ run.sh
data/ # snapshots or synthetic substitutes (never raw victim traffic)
src/ # the code, with the dangerous parts gated (below)
ETHICS.md # disclosure status, gating rationale, usage boundaries
Give evaluators the shortest true path to each result. A "kick-the-tires" phase means the first five minutes decide the evaluator's mood — make the smoke test one command.
Packaging dangerous material responsibly
NDSS artifacts routinely include working exploits, attack tooling, or malware-adjacent code. This is allowed; unbounded release is not.
- Gate the weaponized parts. Ship the analysis and measurement pipeline openly; put live exploit modules behind documented access conditions or a defanged demonstration mode consistent with your Ethics Considerations section.
- Align with the disclosure clock. If a vulnerability is under a coordinated-disclosure embargo, the artifact's release timing must respect it — state the date the full module becomes available.
- Sandbox by default. Anything that touches a network should target a supplied local testbed, not the Internet, out of the box. Document the blast radius.
- No live victim data. Replace with prefix-preserved or synthetic traces carrying the
statistics the result needs (see
ndss-reproducibility).
When re-execution needs your lab
Hardware-bound results (SDRs, specific switches, IoT devices) cannot be reproduced by a committee at their desks. Bridge the gap: provide recorded traces plus a replay harness so the analysis is reproducible even when the capture is not, and say plainly in the README which results are trace-replayable and which need the physical setup. This commonly yields Functional + Reproduced-on-recorded-data rather than a failed full Reproduced.
The 2-page artifact appendix
Written after evaluation for the camera-ready. It states the badges earned, the artifact's scope, access instructions (the DOI), hardware/software requirements, and the mapping from paper claims to artifact components. Keep it factual and navigational — it is a map for future readers of the open-access paper, not a second results section.
Timing
The opt-in comes shortly after (conditional) acceptance and runs on its own schedule (2027 dates 待核实 — confirm on the Call for Artifacts). It overlaps the camera-ready window (Jan 6, 2027), so a team that starts packaging at acceptance, not at the AE deadline, avoids a two-front crunch.
Output format
[Badges targeted] Available / Functional / Reproduced — with feasibility per badge
[Package status] README / INSTALL / per-claim runners / data substitutes — done or gaps
[Danger handling] gated modules, sandbox default, disclosure-aligned release date
[Hardware gap] results needing physical setup + trace-replay bridge
[Appendix] 2-page artifact appendix drafted? claim→component map complete?