brycewang-stanford OSDI Artifact Evaluation
Package the system for the artifact evaluation committee (AEC). Everything
cycle-specific below is OSDI '26 as verified 2026-07-08 — and 2026 changed the rules,
so check the live Call for Artifacts (usenix.org/conference/osdi26/call-for-artifacts
pattern) before optimizing for the wrong target.
How OSDI AE is wired
- Who: an AEC organized with the sysartifacts community
(
sysartifacts.github.io), which runs artifact evaluation across the systems conferences — reviewers are systems students and researchers on machines that are not yours. - When: artifacts are submitted after the paper is (conditionally) accepted — the '26 deadline was May 8, 2026, 8:59 pm PDT, about six weeks after the March 26 notification. The chairs explicitly encouraged preparing the artifact while the paper was still under review; teams that ignored this spent April in a panic.
- Stakes: badges appear on the published paper. AE is optional but has become an expectation for systems papers claiming practical relevance — and it is the prerequisite for the final paper's two-page Artifact Appendix.
The 2026 badge narrowing
| Cycle | Badges evaluated |
|---|---|
| OSDI '21–'25 (sysartifacts calls) | Artifacts Available, Artifacts Functional, Results Reproduced |
| OSDI '26 | Artifacts Available only |
This is the pack's sharpest example of cycle volatility. In 2026 the AEC judged one thing: that the artifacts have been made available for retrieval, permanently and publicly. Zenodo was the encouraged host; institutional repositories and third-party archives (FigShare, Dryad, Software Heritage, GitHub, GitLab) were acceptable.
Two disciplined responses:
- Meet the letter: a permanent, resolvable archive — not a personal homepage, not a repo you might rename. Mint the DOI/permanent identifier before the deadline and put that identifier in the paper.
- Build to the historical bar anyway. Functional/Reproduced may return in later cycles, your readers will attempt reproduction regardless (the proceedings are open access), and a runnable artifact is what makes the badge worth having. The structure below targets reproducibility even when only availability is graded.
Artifact anatomy
osdi26-artifact/ # archived as one versioned unit (e.g. Zenodo)
├── README.md # THE document: claims map + kick-the-tires + full runs
├── LICENSE # explicit; AEC members must be allowed to run it
├── system/ # source at the paper's commit, build instructions
├── baselines/ # versions/tags, tuning configs, rebuild notes
├── workloads/ # traces or regeneration scripts + provenance/licensing
├── experiments/ # one driver per paper claim: run_fig7.sh, run_tab3.sh
├── expected/ # reference outputs + tolerance notes (variance!)
├── ledger/ # provenance records from osdi-reproducibility
└── environment/ # container/VM image or exact setup script; hw requirements
The README's claims map is the heart: a table from paper claim (Fig. 7, Table 3, "recovery under 900 ms") to driver script, expected output, tolerance, and runtime. An evaluator should reach a first signal — the "kick-the-tires" path — in under 30 minutes on commodity hardware, even if headline experiments need a testbed.
Systems-specific honesty
- Hardware walls: if a claim needs 64 nodes or specific NICs, say so at the top and provide a scaled-down mode that exercises every code path with reduced magnitudes; state which numbers the small mode can and cannot reproduce.
- Variance: publish tolerance windows per claim ("p99 within ±10% across 10 runs"), sourced from your own ledger — an evaluator who gets 3.1x where the paper says 3.2x should find reassurance, not confusion.
- Licensing traps: production-derived traces often cannot be redistributed; include the generator plus provenance documentation, and say so plainly rather than shipping a mystery synthetic substitute.
- De-anonymization timing: AE runs post-acceptance, so the artifact need not be anonymous in 2026's flow — but verify this against the current call rather than assuming it, since submission-time AE exists at other systems venues.
Working with the AEC
Artifact evaluation is a cooperative review, and the committee's constraints shape what a good artifact looks like:
- Evaluators are typically graduate students and postdocs handling several artifacts in a fixed window — assume limited time per artifact and front-load the payoff: the fastest path from download to a verifiable signal wins goodwill that carries through the harder experiments.
- Expect a communication channel (the AE submission system) for questions during evaluation; answer fast and patch the README rather than replying inline, so the archived artifact ends evaluation better than it started. Where the archive is immutable (a minted DOI version), publish a new version rather than editing history.
- Provide fallbacks per failure point: a prebuilt container beside the build-from-source path, a small dataset beside the trace generator, expected logs beside every driver so a partial failure is diagnosable from output alone.
- If the badge set is availability-only (as in 2026), evaluators still need to retrieve and inspect the archive — a 40 GB tarball with no manifest can fail even that bar in practice. Ship a top-level manifest with sizes and checksums.
From badge to Artifact Appendix
Papers passing evaluation may add up to two pages of Artifact Appendix to the
final paper (June 9 deadline, osdi-camera-ready): a standard-format description of
scope, requirements, and reproduction steps. Write it by condensing the README the
AEC just validated — it is the only reviewed-adjacent supplement OSDI has, and it
outlives the conference as the artifact's citable front door.
Output format
[Cycle badge set] confirmed from live Call for Artifacts: <list> (待核实 if unread)
[Archive] permanent host + identifier minted? <where / missing>
[Claims map] paper claims covered by drivers: <n/m>; kick-the-tires < 30 min? yes/no
[Hardware honesty] testbed requirements + scaled-down mode documented? yes/no
[Timeline] days to artifact deadline; README/expected-output gaps: <list>