brycewang-stanford CoNEXT Artifact Evaluation
Turn the accepted paper's package into optional ACM reproducibility badges, administered by CoNEXT's reproducibility committee. This is a distinct process from the paper review, with its own timing: the decisive move — opting in before the paper submission deadline — happens long before you have an accept. Treat badging as a track you enter at submission and complete after acceptance.
The badge timeline (opt-in is the gate)
[Before submission deadline] OPT IN for ACM badging <-- miss this and you are ineligible this cycle
[Within ~1 week of accept] send a ONE-PAGE artifact description + pointers to the committee
[Post-accept evaluation] committee reviews the artifact against the ACM badge criteria
[Camera-ready] approved badges appear on the PACMNET article (see conext-camera-ready)
The ACM badges (what each requires)
CoNEXT uses the ACM Artifact Review and Badging scheme. In practice the ladder is:
| Badge | What it certifies | What you must provide |
|---|---|---|
| Artifacts Available | The artifact is publicly archived | A DOI-issuing archive (Zenodo/figshare/Software Heritage) with an open license |
| Artifacts Evaluated — Functional | It is documented, complete, and runs | Clear docs, a runnable package, and a "getting started" path that works |
| Artifacts Evaluated — Reusable | It is functional and others can build on it | Reusability docs, clean structure, parameterization, and a license that permits reuse |
| Results Reproduced | Key results were regenerated by evaluators | A scripted path that regenerates the paper's central figures/tables from the inputs |
Confirm the exact badge set and names offered by the current CoNEXT reproducibility committee — the scheme is stable but wording is cycle-volatile (待核实).
The one-page artifact description
The committee starts from your one-pager, so make it do work:
- Inventory: code, traces, configs, testbed scripts — with a pointer to each.
- Requirements: the hardware/software an evaluator needs. Networking artifacts often need specific hardware (a particular switch ASIC, NIC, or SDR); if evaluators need access to hardware you must provide, flag it prominently and describe how you will give access.
- Claims supported: which paper results the artifact reproduces, and which it cannot (be honest about hardware-bound results).
- Getting started: the shortest path to a first successful run.
Networking-specific evaluation hurdles
- Hardware dependence. A result that needs a specific switch or NIC is hard for a remote evaluator; provide a testbed reservation, a remote-access path, or a faithful emulation for the parts that can run in software, and clearly mark what is hardware-bound.
- Trace size and privacy. Large or sensitive traces may not be fully releasable; ship a representative or synthetic subset plus the pipeline, and document the restriction.
- Environment drift. Pin firmware/OS/kernel versions and provide images; "works on our testbed" is not reproducible.
- Live dependencies. Cache anything that would otherwise require a live API/network call so the evaluator's run is deterministic.
Reusability that survives an evaluator
- A README that assumes nothing about your lab's setup; every path and dependency explicit.
- A single command (or a short script) for the "getting started" run, and a documented full-reproduction path for the central results.
- An open license present and consistent across the archive.
- A mapping from paper claim → artifact step → expected output, so an evaluator can check "Results Reproduced" without guessing.
Pre-evaluation checklist
[Opt-in] confirmed before the submission deadline? yes/no
[Archive] DOI-issuing archive + open license? yes/no
[One-pager] inventory + requirements + hardware-access plan + claims map? yes/no
[Functional] fresh-environment run succeeds from the docs alone? yes/no
[Reusable] structure + docs let a third party adapt it? yes/no
[Reproduced] central figures/tables regenerate by script? yes/no
[Hardware] hardware-bound results flagged; access path provided? yes/no
Output format
[Badge target] Available / Functional / Reusable / Reproduced (which are in scope)
[Opt-in status] done before submission? yes/no
[Artifact readiness] <per-badge gaps>
[Hardware plan] <how remote evaluators reach any required hardware>
[One-pager] <drafted / sent to committee>