Osdi Artifact Evaluation

OSDI Artifact Evaluation

Package the system for the artifact evaluation committee (AEC). Everything cycle-specific below is OSDI '26 as verified 2026-07-08 — and 2026 changed the rules, so check the live Call for Artifacts (usenix.org/conference/osdi26/call-for-artifacts pattern) before optimizing for the wrong target.

How OSDI AE is wired

• Who: an AEC organized with the sysartifacts community (sysartifacts.github.io), which runs artifact evaluation across the systems conferences — reviewers are systems students and researchers on machines that are not yours.
• When: artifacts are submitted after the paper is (conditionally) accepted — the '26 deadline was May 8, 2026, 8:59 pm PDT, about six weeks after the March 26 notification. The chairs explicitly encouraged preparing the artifact while the paper was still under review; teams that ignored this spent April in a panic.
• Stakes: badges appear on the published paper. AE is optional but has become an expectation for systems papers claiming practical relevance — and it is the prerequisite for the final paper's two-page Artifact Appendix.

The 2026 badge narrowing

This is the pack's sharpest example of cycle volatility. In 2026 the AEC judged one thing: that the artifacts have been made available for retrieval, permanently and publicly. Zenodo was the encouraged host; institutional repositories and third-party archives (FigShare, Dryad, Software Heritage, GitHub, GitLab) were acceptable.

Two disciplined responses:

1. Meet the letter: a permanent, resolvable archive — not a personal homepage, not a repo you might rename. Mint the DOI/permanent identifier before the deadline and put that identifier in the paper.
2. Build to the historical bar anyway. Functional/Reproduced may return in later cycles, your readers will attempt reproduction regardless (the proceedings are open access), and a runnable artifact is what makes the badge worth having. The structure below targets reproducibility even when only availability is graded.

Artifact anatomy

osdi26-artifact/                    # archived as one versioned unit (e.g. Zenodo)
├── README.md                       # THE document: claims map + kick-the-tires + full runs
├── LICENSE                         # explicit; AEC members must be allowed to run it
├── system/                         # source at the paper's commit, build instructions
├── baselines/                      # versions/tags, tuning configs, rebuild notes
├── workloads/                      # traces or regeneration scripts + provenance/licensing
├── experiments/                    # one driver per paper claim: run_fig7.sh, run_tab3.sh
├── expected/                       # reference outputs + tolerance notes (variance!)
├── ledger/                         # provenance records from osdi-reproducibility
└── environment/                    # container/VM image or exact setup script; hw requirements

The README's claims map is the heart: a table from paper claim (Fig. 7, Table 3, "recovery under 900 ms") to driver script, expected output, tolerance, and runtime. An evaluator should reach a first signal — the "kick-the-tires" path — in under 30 minutes on commodity hardware, even if headline experiments need a testbed.

Systems-specific honesty

• Hardware walls: if a claim needs 64 nodes or specific NICs, say so at the top and provide a scaled-down mode that exercises every code path with reduced magnitudes; state which numbers the small mode can and cannot reproduce.
• Variance: publish tolerance windows per claim ("p99 within ±10% across 10 runs"), sourced from your own ledger — an evaluator who gets 3.1x where the paper says 3.2x should find reassurance, not confusion.
• Licensing traps: production-derived traces often cannot be redistributed; include the generator plus provenance documentation, and say so plainly rather than shipping a mystery synthetic substitute.
• De-anonymization timing: AE runs post-acceptance, so the artifact need not be anonymous in 2026's flow — but verify this against the current call rather than assuming it, since submission-time AE exists at other systems venues.

Working with the AEC

Artifact evaluation is a cooperative review, and the committee's constraints shape what a good artifact looks like:

• Evaluators are typically graduate students and postdocs handling several artifacts in a fixed window — assume limited time per artifact and front-load the payoff: the fastest path from download to a verifiable signal wins goodwill that carries through the harder experiments.
• Expect a communication channel (the AE submission system) for questions during evaluation; answer fast and patch the README rather than replying inline, so the archived artifact ends evaluation better than it started. Where the archive is immutable (a minted DOI version), publish a new version rather than editing history.
• Provide fallbacks per failure point: a prebuilt container beside the build-from-source path, a small dataset beside the trace generator, expected logs beside every driver so a partial failure is diagnosable from output alone.
• If the badge set is availability-only (as in 2026), evaluators still need to retrieve and inspect the archive — a 40 GB tarball with no manifest can fail even that bar in practice. Ship a top-level manifest with sizes and checksums.

From badge to Artifact Appendix

Papers passing evaluation may add up to two pages of Artifact Appendix to the final paper (June 9 deadline, osdi-camera-ready): a standard-format description of scope, requirements, and reproduction steps. Write it by condensing the README the AEC just validated — it is the only reviewed-adjacent supplement OSDI has, and it outlives the conference as the artifact's citable front door.

Output format

[Cycle badge set] confirmed from live Call for Artifacts: <list> (待核实 if unread)
[Archive] permanent host + identifier minted? <where / missing>
[Claims map] paper claims covered by drivers: <n/m>; kick-the-tires < 30 min? yes/no
[Hardware honesty] testbed requirements + scaled-down mode documented? yes/no
[Timeline] days to artifact deadline; README/expected-output gaps: <list>

Source: brycewang-stanford/Awesome-Journal-Skills → OSDI-Skills/skills/osdi-artifact-evaluation/SKILL.md