brycewang-stanford OOPSLA Artifact Evaluation
SPLASH runs a unified artifact-evaluation track for its research papers; the 2026 edition used a two-phase process — an opening kick-the-tires pass where evaluators check the artifact builds and starts at all, then the full evaluation — awarding Functional (documented, complete enough to exercise), Reusable (Functional plus organization and documentation that support reuse by others), and Available (a snapshot deposited on Zenodo with a DOI), with Available strongly encouraged for every passing artifact absent licensing or privacy constraints. Per-cycle dates and submission mechanics live on the current track page (待核实 each cycle).
Design for the kick-the-tires failure mode
Most artifacts that miss a badge die in the first hour of a stranger's time: a build that assumes your machine. The discipline is to treat the evaluator as a hostile fresh VM.
Golden-path contract (put this at the top of README.md):
1. Requirements: OS/arch, RAM, disk, expected wall-clock time
2. One command to build (container image or pinned toolchain)
3. One command for a 10-minute smoke result mapped to a named claim
4. One command per paper table/figure, each with expected output range
5. What may legitimately differ on other hardware, and by how much
Dry-run the contract yourself on a machine that has never seen the project — laptop-hosted paths, private registries, and license-gated dependencies are the classic silent breakers.
Badge strategy
| Badge | What actually earns it | Cheap mistakes that forfeit it |
|---|---|---|
| Functional | Inventory + docs sufficient to exercise the artifact; results consistent with the paper | Missing inputs; undocumented flags; claims that need hardware you didn't disclose |
| Reusable | Clear structure, extension points, explained internals — a stranger could build on it | A working but opaque tarball; hard-coded paths; no guidance beyond replication |
| Available | Zenodo (or equivalent archival) deposit with DOI | GitHub-only "archive" (mutable, not archival); DOI minted after the camera-ready statement was frozen |
Reusable is where OOPSLA-style artifacts differentiate: language implementations, calculi mechanizations, and corpus studies are exactly the artifacts other groups extend, so structure the repository as a tool, not as a paper appendix.
Claim-to-artifact mapping
Evaluators read the paper's claims against what the artifact demonstrates.
Build the mapping explicitly and reuse it in three places: the artifact
README, the evaluation submission form, and the paper's Data-Availability
Statement (oopsla-submission requires the statement; oopsla-camera-ready
updates it with the final DOI).
- Every quantitative table/figure → a script that regenerates it.
- Every qualitative claim ("scales to", "handles all of") → a named test or corpus directory that witnesses it.
- Every claim the artifact cannot support (proprietary benchmark, cluster scale) → an explicit exclusion with justification, declared up front rather than discovered by the evaluator.
Scope and timing notes
- Artifact evaluation is tied to acceptance rounds; papers accepted in Round 1 and Round 2 flow through the process in different windows — read the track page for the round you were accepted in.
- Zenodo deposits are versioned: mint the DOI early, then publish updated versions as evaluation feedback lands; the DOI in the published article should resolve to the final version.
- Anonymity is over at this stage, but the paper's review-time supplement
and the artifact must not contradict each other (
oopsla-supplementary).
Output format
[Phase readiness] kick-the-tires dry-run: pass / failures listed
[Badge targets] Functional / +Reusable / +Available, with gaps per badge
[Claim map] <n claims mapped / m unmapped — list unmapped>
[Deposit] Zenodo DOI status + version plan
[Statement sync] Data-Availability Statement consistent: yes/no