brycewang-stanford FAST Artifact Evaluation
Use this for the artifact track. FAST follows the USENIX artifact-evaluation model, and evaluation is a separate, post-acceptance process with its own timeline and an Artifact Evaluation Committee (AEC). Two things to internalize: badges are earned by evaluators actually using your package, and the review-time artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived, with an artifact appendix).
The three USENIX badges (verify the current set and names)
| Badge | What it certifies | What earns it |
|---|---|---|
| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |
| Artifacts Functional | The artifact works and is useful for the paper's outcomes | A documented build/run, a demo, and expected outputs on the AEC's setup |
| Results Reproduced | An evaluator reproduced the paper's main results | A path from the artifact to the headline numbers |
Available is a low-cost, high-value badge (archive the package). Functional and Reproduced require the evaluator's own run to succeed — and for a storage paper that run may need specific hardware, which shapes everything below.
The storage-artifact problem: hardware dependence
A storage result often depends on a particular SSD/NVM device, firmware, or a large trace. Design the package so an AEC can get as far as possible without your exact drive:
[Tier the claims] which results are hardware-agnostic (analysis, plots from logs) vs. device-bound
(measured WA, tail latency)? Make the agnostic ones turnkey.
[Ship the logs] include the raw device-counter logs and the scripts that turn them into figures,
so "Reproduced" is achievable from data even when the device is unavailable
[Document devices] state exact models + firmware; where feasible, offer a smaller commodity-SSD path
the AEC can run to see the trend, and say the factor is device-specific
[Bound the runtime] long endurance/aging runs: provide a short demo + the full slow protocol, and
label which results need the full run
What a storage AEC opens first
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| A storage system/tool | The README and one build+run command | Undocumented deps; only-builds-on-authors'-kernel |
| A measured storage cost (WA/latency) | The logs + the script from logs to the figure | Numbers in the PDF no script regenerates |
| A trace-driven study | The replay tool + the archived trace | Trace named, not shipped; replay settings missing |
| A crash-consistency claim | The fault-injection / replay harness | Only the FS code, no way to reproduce the test |
| A reliability field study | The (de-identified) dataset + analysis scripts | Aggregates only; nothing an evaluator can re-run |
Assume an evaluator gives your package a bounded time budget. Design for the first ten minutes (build
- a small demo + a figure-from-logs) to succeed.
Packaging plan
[Container] ship a Dockerfile or a pinned environment; avoid "install these 40 things by hand"
[README] one screen: what it is, build, run the demo, reproduce each claim, expected runtime/outputs
[Mapping] an explicit table: paper claim -> script -> expected result (mark device-bound ones)
[Data] the archived traces and raw device-counter logs, not just names
[Provenance] device models + firmware, host/kernel, mkfs/mount options, seeds
[License] an OSI-approved license so the artifact can be evaluated toward Functional/Reproduced
[Archive] deposit in a DOI-issuing repository for the Available badge
[Appendix] write the artifact appendix; add the earned badges to the camera-ready appendix after AE
Anonymized review artifact vs. badge artifact
- At submission: anonymized for the paper's reviewers — no owner strings, hostnames, cluster paths, serials in device dumps, lab names, or identity-revealing trace hosts.
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; write the artifact appendix; this is the version the AEC badges and the camera-ready cites, with badges shown on the first page.
Worked vignette: packaging a KV-store + measurement study
A paper contributes an SSD-conscious KV store and a write-amplification study. To target Reproduced:
ship a Docker image with the store pre-built; a run_demo.sh that runs a short YCSB load on a
commodity SSD (or a loopback file) in minutes; a reproduce/ directory whose scripts regenerate each
figure from the shipped device-counter logs so the headline numbers reproduce even without the
exact datacenter SSD; a claim-to-script table marking which results are device-bound; the archived
trace with replay scripts; and an MIT/Apache license. State honestly which numbers are the specific
drive's and which are the trend.
Calibration
- The artifact timeline is after acceptance and independent of the camera-ready deadline; do not conflate them.
- Badge names, the exact set offered, and whether AE is single- or double-anonymous vary by cycle (待核实 for the precise FAST '27 AEC anonymity model) — confirm on the current Call for Artifacts.
Output format
[Target badges] Available / Functional / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <system/traces/device-logs/scripts/provenance/license>
[Hardware tiering] <which claims turnkey-from-logs vs. device-bound; demo path present?>
[Ten-minute test] does build + demo + a figure-from-logs succeed on a clean machine? yes/no
[Fixes before upload] <ordered list>