brycewang-stanford MLSys Artifact Evaluation
Use this after acceptance, when deciding whether and how to enter MLSys artifact evaluation. AE is one of this venue's defining institutions: a separate committee evaluates how well artifacts support the paper's claims and awards badges. In the 2026 cycle (verified 2026-07-08): submission of artifact abstract, paper PDF, and Artifact Appendix to a dedicated AE site by March 8; evaluation window March 8 - April 8; badges for Availability, Functional, and Reproducible; anonymous reviewer-author interaction during evaluation; and a small number of Distinguished Artifact Awards for exceptional packages. Reopen the current Call for Artifact Evaluations before relying on any of these mechanics.
Badge strategy — decide the target first
| Badge | What evaluators check | Typical cost to authors |
|---|---|---|
| Availability | Artifact is publicly and permanently retrievable (public GitHub-style link expected in the appendix) | Hours: license, public repo, archival snapshot |
| Functional | The artifact runs: documented, complete relative to the paper, exercisable end-to-end | Days: install path, smoke test, small-scale example |
| Reproducible | Evaluators regenerate the paper's key results following your instructions | Weeks: automation, hardware plan, tolerance definitions |
Claim badges honestly. Requesting Reproducible when the headline table needs a cluster the committee cannot access converts a friendly process into a failed one; requesting only Availability+Functional with a clear statement of why is respected.
The hardware problem — MLSys AE's hardest part
Most MLSys results are performance results on specific hardware. Solve this explicitly in the appendix rather than hoping:
- Tier the claims. Separate results reproducible on one commodity GPU (or CPU) from results needing an 8-GPU node from results needing a cluster. Ask for the Reproducible badge on the tiers evaluators can actually run.
- Scale-down targets. Provide a reduced configuration (smaller model, shorter trace) whose trend matches the paper — and state the expected numbers for that reduced configuration, since evaluators cannot compare against a table they cannot reproduce.
- Relative, not absolute, tolerances. Performance varies across machines; define success as "speedup over the included baseline within ±15%" rather than absolute throughput.
- If you can offer supervised access to your hardware, check whether the current AE call permits it; do not assume.
Package skeleton evaluators expect
# Dockerfile — pin the system layer, not just Python
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04
RUN apt-get update && apt-get install -y git python3.11 python3-pip
COPY requirements.lock /w/requirements.lock # exact versions, hash-pinned
RUN pip install --no-cache-dir -r /w/requirements.lock
COPY . /w
WORKDIR /w
# One command per claim tier:
# make smoke (<10 min, any GPU) -> Functional
# make table3 (~1 h, 1x A100/H100) -> Reproducible tier 1
# make fig6-scaled (reduced trace, expected: 1.5-1.7x over baseline)
CMD ["make", "smoke"]
- README with: claims-to-commands map, hardware/time requirements per command, and a "what should I see" block for every command.
- All baselines included at the versions and configurations used in the paper — a package that reproduces your system but not the comparison reproduces nothing.
- Logged outputs from your own runs, so evaluators can diff behavior before burning GPU-hours.
- Traces, datasets, or generators for the workloads; if a workload is proprietary, ship a synthetic surrogate and label the substitution honestly.
Artifact Appendix skeleton
The appendix is the evaluator's map; in 2026 it was submitted with the artifact abstract and paper PDF to the AE site. Whatever template the current call mandates, cover:
A.1 Abstract what the artifact contains, one paragraph
A.2 Claims supported paper claim -> command -> expected output -> tolerance
A.3 Requirements hardware (per tier), software, disk, network access,
time-to-first-result and time-to-full-reproduction
A.4 Setup container build or install path; offline fallback for
anything fetched from the network at build time
A.5 Experiment map make targets / scripts per figure and table number
A.6 Known deviations results that vary by hardware and by how much;
proprietary workloads replaced by surrogates
A.7 Reusability how to point the artifact at new models/workloads —
this is what separates award-level artifacts
Write A.3 pessimistically: an evaluator who discovers an undeclared 200GB download or a hidden internet dependency mid-window will not restart with goodwill.
During the evaluation window
- Interaction is anonymous and mediated; respond within a day — the window is finite (one month in 2026) and a stuck evaluator is a failed badge.
- Treat every evaluator failure as a packaging bug to fix and re-push, not a user error to explain away; committees typically allow artifact updates during evaluation.
- Keep one author on call who can debug environment issues; the most common failure mode is driver/container mismatch on the evaluator's machine, not your code.
Why bother
Badges appear with the paper and signal to this community — where results are routinely questioned as hardware-specific — that a third party ran your code. The Distinguished Artifact Award exists because MLSys treats the artifact as part of the scholarship, and a badged artifact keeps producing citations after the conference ends.
Cycle-volatility warnings
- Badge names, the AE site, deadlines, and whether AE remains post-acceptance-optional are all per-cycle decisions; the 2026 mechanics above are anchors to verify.
- The Artifact Appendix template, if one is mandated, comes from the current AE call.
Output format
[Badges targeted] availability / +functional / +reproducible (per claim tier)
[Claim tiers] <claim -> hardware needed -> reproducible by AE? >
[Package status] <docker/README/baselines/workloads/logged-outputs>
[Scale-down plan] <reduced configs + expected numbers>
[On-call owner] <person for the evaluation window>
[Gaps before AE deadline] <ordered list>