Asplos Artifact Evaluation

ASPLOS Artifact Evaluation

Artifact evaluation at ASPLOS is a post-acceptance, opt-in, collaborative process: an independent committee works with authors to validate the paper's key results, and successful artifacts carry badges on the published paper (AE pages, checked 2026-07-08). It is also a tradition the venue itself highlights — systems readers increasingly treat an unbadged systems paper as a weaker citation. Treat AE as part of the publication, budgeted like a small sixth section.

The three badges and what each actually demands

Evaluators assign scores per requested badge and record what they could and could not reproduce — so the artifact's job is to make their success path short and their failure modes diagnosable.

The Artifact Appendix is the contract

ASPLOS 2027 expects an Artifact Appendix built from the provided ae.tex template (or equivalent sections) covering: all software, hardware, and dataset dependencies; the key results to be reproduced; and how to prepare, run, and validate the experiments. Write it as if the evaluator is competent, busy, and using different hardware than yours:

• Dependencies include the awkward ones: kernel versions, privileged access, BIOS settings, board models, expander firmware — everything from the state ledger in asplos-reproducibility.
• "Key results" means a selected subset: pick the claims that define the paper, not all 40 bars of every figure. Ambition here creates failure reports.
• Validation must be decidable: state the expected output and the tolerance within which the claim holds ("ordering preserved; absolute times ±15%").

Package layout that evaluators can navigate blind

artifact/
  README.md            # 10-minute quick start + full map
  APPENDIX.pdf         # the ae.tex appendix as submitted
  env/                 # container/VM recipe OR exact install script
  hardware.md          # tiered requirements + what to do without them
  run/
    smoke.sh           # minutes-scale end-to-end sanity check
    claim1_fig6.sh     # one script per key result, named for its figure
    claim2_tab3.sh
  expected/            # reference outputs + tolerance statement per claim
  data/ or data.md     # datasets, or archival pointers + checksums

Hardware-dependent claims: give evaluators a path

The recurring ASPLOS AE failure is a paper whose headline number needs silicon the committee lacks. Acceptable mitigations, in descending order of strength:

1. Provide remote access to the platform for the evaluation window (with an anonymity-safe access route if the process requires it).
2. Ship the simulator-backed subset as the reproducible core, and mark the silicon results as demonstrably-run (logs + analysis pipeline included).
3. Offer a scaled-down proxy (smaller FPGA, reduced workload) with an explicit argument for why the proxy's behavior transfers.

Say which mitigation applies in the appendix, per claim — evaluators score against what you requested, so calibrated requests outperform hopeful ones.

Collaboration protocol

• Expect rounds: evaluators report blockers, authors fix and respond. Reserve maintainer time in the weeks after camera-ready (exact 2027 AE dates: 待核实 — confirm at notification).
• Fix-forward, do not re-argue: an evaluator's confusion is a defect in the README.
• Keep the artifact frozen at a tagged version during evaluation; hotfixes go on a branch the evaluators are told about.

Common evaluator blockers, pre-empted

Field experience across systems AE committees converges on a short list of first-hour failures, all preventable:

• Undeclared credentials or licenses — a workload, simulator model, or dataset that needs a registration the evaluator lacks; declare it in the appendix and provide an alternative path.
• Hidden network assumptions — builds that fetch from internal mirrors or rate-limited hosts; vendor the dependencies or provide the container image.
• Root-only steps without warning — kernel-module or BIOS-adjacent steps must be flagged up front so the evaluator can pick a sacrificial machine.
• Hour-scale first feedback — if the smoke test takes an evening, the first blocker report costs a full round trip; minutes-scale smoke tests keep the collaboration inside the calendar.
• Output the evaluator must interpret — raw logs with no comparator; every claim script should end by printing PASS/FAIL against the tolerance.

Anonymity boundary

AE runs after acceptance, so evaluator-facing materials need not be anonymous — but any artifact pointer placed in the submission itself (an appendix teaser, a footnoted repository) falls under the double-blind rules and must be anonymized end to end: repository owner, commit author strings, container registry paths, and dataset hosting all leak identity. The clean pattern is to keep the submission's artifact story descriptive ("we will submit an artifact covering claims 1-3") and materialize the links only in the Artifact Appendix after notification.

Dry-run protocol

Before submission to the AEC, have a colleague who did not build the artifact execute the README on a clean machine, timing each stage and noting every question they had to ask. Their questions are defects; fix the README, not the colleague. Two such passes typically halve the evaluation rounds.

Output format

[Badges requested] available / functional / reproducible — with rationale
[Appendix status] dependencies / key results / prepare-run-validate all drafted: Y/N
[Smoke test] clean-environment runtime: N min · passes: Y/N
[Claim scripts] one per key result, mapped to figure/table: list
[Hardware path] per silicon-dependent claim: access / sim-subset / proxy
[Archive] DOI-issuing repository chosen + deposit dry-run done: Y/N

Source: brycewang-stanford/Awesome-Journal-Skills → ASPLOS-Skills/skills/asplos-artifact-evaluation/SKILL.md