brycewang-stanford USENIX Security Artifact Evaluation
USENIX Security '26 split artifact evaluation into two phases with different stakes:
Phase-1 (availability) is mandatory and paper acceptance is conditional on it,
running between acceptance and the final-paper deadline; Phase-2 (functionality
and reproducibility) is optional, running after finals are due. The community
process and submission mechanics live at secartifacts.github.io/usenixsec2026/;
the venue framing is on the usenix.org Call for Artifacts page (both checked
2026-07-08, via search renderings since usenix.org direct fetch returned 403).
The three badges
| Badge | What the AEC verifies | Phase |
|---|---|---|
| Artifacts Available | The artifacts named in the Open Science appendix exist at the stated location and are retrievable | Phase-1, required of all accepted papers (or a justified omission) |
| Artifacts Functional | Core functionality works as documented: installs, runs, produces sane output | Phase-2, opt-in |
| Results Reproduced | The paper's reported results were regenerated by the AEC | Phase-2, opt-in |
Badges are issued independently — Reproduced does not require re-litigating Available. The strategic read: Phase-1 is table stakes; Phase-2 is where a systems or measurement paper earns durable credibility (and eligibility for artifact recognition such as the distinguished-artifact honors given in recent years).
Phase-1: make "available" mean available
The check is simple and unforgiving — an evaluator follows the pointer in your Open Science appendix. Failure modes are all self-inflicted:
- The submission-time anonymous mirror was never replaced by a permanent home.
- The repository exists but the named tag/commit does not, or the dataset link requires a lab account.
- The appendix says "code and data" but the archive holds code only.
Use an archival service that mints DOIs (e.g., Zenodo) or an institutional archive
for the frozen version, plus a living repository for maintenance. If something
genuinely cannot be released — live vulnerability details under embargo, PII-laden
measurement data, malware corpora with legal constraints — the appendix must say
so and say why; justified withholding is policy-compliant, silent withholding is
not (see usenixsec-reproducibility for the decision table).
Phase-2: security artifacts have special failure modes
Evaluators are volunteers on ordinary machines and a clock. Security artifacts add hazards other fields do not have — design these out:
- Dangerous-by-default behavior. An exploit PoC or scanner must ship pointed at a local target (container victim, testbed config), never at the live internet. Document the blast radius of every script.
- Environment brittleness. Kernel-version-sensitive attacks, specific GPU drivers, SGX hardware: state exact requirements up front and provide a VM or container image when the host matters.
- Nondeterministic attacks. Success-rate claims need the trial count and the tolerance you expect the evaluator to observe, not a single lucky run.
- Long horizons. If full reproduction takes days (fuzzing campaigns, large scans), provide a scaled-down kit that reproduces the trend in under an hour, plus the full recipe.
artifact/
├── README.md # claims ↔ experiments map; time + hardware budget
├── LICENSE
├── Dockerfile # or VM image link with checksum
├── setup.sh # one-shot environment build (prints versions)
├── run_minimal.sh # <1h: reproduces Fig 5 trend on the local testbed
├── run_full.sh # complete campaign; documents multi-day runtime
├── victim/ # self-contained target — nothing external is probed
├── data/ # released datasets + SHA256SUMS; withheld items listed
└── expected/ # reference outputs with tolerance notes per claim
The README's spine should be a claims table: each major claim in the paper mapped to the command that regenerates it, its runtime, and the expected output range. Write for an evaluator who reads the paper once and your README three times.
Working the process
- Register the artifact by the AE deadline for your cycle (dates on the secartifacts instructions page — per-cycle, 待核实 each time).
- During evaluation, communication runs anonymously through the AE platform; respond fast, since the window is short and iterations are the norm.
- Fix-and-rerun is expected: evaluators report blockers, authors patch. Budget maintainer attention for the whole window, not just the submission day.
Reverify each cycle
- Whether Phase-1 remains mandatory and acceptance-conditional in the current CFP.
- Badge definitions and AE deadlines on the current secartifacts instance.
- Size/hosting norms and whether the AEC supplies compute for special hardware.
Output format
[Phase-1] artifact home (DOI/URL) + retrievability test result; omissions justified
[Phase-2 target] none / Functional / Reproduced — with reasoning
[Hazard review] network side effects, secrets, licensing of third-party payloads
[Kit status] minimal-run time, full-run time, claims table coverage n/m
[Deadlines] AE registration + evaluation window for this cycle