FAST Reproducibility
Use this before submission and again before camera-ready. FAST's storage results live on real
hardware that ages, throttles, and varies part-to-part, so reproducibility here is a distinct
craft from software-only venues: a reader reproducing your work needs to know not just the code but
the device, its firmware, and its state. The goal is that a competent reader with comparable
hardware could rebuild your evidence and reach your conclusions.
Evidence map
- Map each claim and reported number to a verifiable location — a paper section, a table
generated from a logged run, or a script in the artifact.
- For a system, give enough of the design, parameters, mkfs/mount options, and build environment
that a reader could rebuild and run it.
- For measurements, report the device provenance (models, firmware, interface), the host
(CPU, RAM, kernel), and the device state (steady-state/aged, fill, TRIM) — the storage-only
provenance that software artifacts omit.
- Keep the availability statement truthful and specific: what code and traces are shared, where
they will live after acceptance, and — if something cannot be shared — exactly why.
- Keep the paper and the artifact consistent: a bytes-written or latency number in the PDF that
no script in the artifact regenerates is the contradiction reviewers read as carelessness.
Availability statement audit
| Claim in the paper |
Weak answer |
FAST-ready answer |
| "We evaluate on SSDs A, B, C" |
"Standard SSDs" |
Exact models, capacities, and firmware versions in a testbed table |
| "Driven by trace T" |
Trace named, not shared |
Archived trace (or documented access) + the replay script and its settings |
| "We reduce write amplification" |
Estimated in prose |
Device-counter (SMART/log) dumps + the script that computes WA from them |
| "Steady-state results" |
Unstated preconditioning |
The preconditioning/aging protocol as a runnable script |
| "Our system is available" |
"Code on request" |
Anonymized, buildable code with a README and a small runnable demo |
"Available on request" is treated as not available; convert every such line into a concrete,
anonymized artifact or an explicit, justified exception (e.g. drives under NDA, privacy-limited
production traces).
Provenance pinning
[Devices] model, capacity, interface, FIRMWARE version; host CPU/RAM, kernel, mkfs/mount options
[State] preconditioning/aging protocol, fill level, TRIM/discard; FOB vs. steady-state disclosed
[Traces] archive the replayed trace or document access; ship the replay tool + timing settings
[Workloads] YCSB/filebench/fio job files with exact parameters and seeds
[Compute] run duration and repeats; note thermal/throttling conditions that affect timing
[Counters] how bytes-written / WA / GC were read from the device, so a reader can re-derive them
Degrees of reproducibility (state the one you achieved)
- Turnkey: one documented command regenerates a table/figure from logged data (best for
analysis and plots).
- Scripted-on-hardware: scripts run end-to-end but require comparable devices and time
(endurance runs, aging, long trace replays); document the hardware and expected runtime.
- Descriptive: prose and provenance detailed enough that a reader with the right hardware could
rebuild the pipeline (acceptable for results that need specific or proprietary devices).
For FAST, aim turnkey for anything that runs from logged data, and be honest that device-bound
results are scripted-on-hardware: state which drives, which firmware, and how long. Promising
turnkey behavior that silently needs a specific SSD is worse than stating the hardware dependency.
The hardware-variance caveat (state it plainly)
Storage numbers are device- and firmware-specific; a reader on a different drive should expect the
trend, not the exact factor. Say so, report per device rather than one blended number where it
matters, and bound generalization — this is not a weakness to hide but the honest storage posture
reviewers reward.
Vignette: a file-system aging study
Consider a study whose results depend on a fragmented, aged volume. Its reproducibility spine: the
aging workload as a runnable script (so the fragmentation state is reconstructable), the exact
mkfs/mount options and kernel, the device models and firmware, the measurement scripts that read
from the device and file-system counters, and the analysis notebooks that turn logs into the paper's
figures — plus one honest sentence about which numbers require the specific drives used.
Consistency and camera-ready pass
- Before submission: every scored number traces to the artifact; the availability statement matches
reality; the artifact is anonymized (no owner strings, hostnames, cluster paths, or personal trace
URLs).
- Before camera-ready: swap anonymized links for a permanent, licensed, DOI-issuing archive
(Zenodo/figshare/Software Heritage) and align the statement with the USENIX artifact badges you are
pursuing (
fast-artifact-evaluation).
Output format
[Claim inventory] <claim -> evidence location>
[Availability] concrete / vague / missing
[Provenance gaps] <device+firmware / device state / trace archival / counters / seeds>
[Reproducibility level] turnkey / scripted-on-hardware / descriptive, stated honestly
[Hardware dependency] <which results need which drives/firmware, stated?>
[Paper fixes] <must appear in the PDF>
[Artifact fixes] <additions before upload>
1---2name: fast-reproducibility3description: Use when strengthening USENIX FAST reproducibility and open-science evidence, covering device and firmware provenance, device-state disclosure, trace availability and replay, claim-to-evidence mapping, honest degrees of reproducibility on hardware that ages and varies, and consistency between what the paper says and what the artifact contains.4---56# FAST Reproducibility78Use this before submission and again before camera-ready. FAST's storage results live on **real9hardware that ages, throttles, and varies part-to-part**, so reproducibility here is a distinct10craft from software-only venues: a reader reproducing your work needs to know not just the code but11the *device, its firmware, and its state*. The goal is that a competent reader with comparable12hardware could rebuild your evidence and reach your conclusions.1314## Evidence map1516- Map each claim and reported number to a **verifiable location** — a paper section, a table17 generated from a logged run, or a script in the artifact.18- For a system, give enough of the design, parameters, mkfs/mount options, and build environment19 that a reader could rebuild and run it.20- For measurements, report the **device provenance** (models, firmware, interface), the **host**21 (CPU, RAM, kernel), and the **device state** (steady-state/aged, fill, TRIM) — the storage-only22 provenance that software artifacts omit.23- Keep the **availability statement** truthful and specific: what code and traces are shared, where24 they will live after acceptance, and — if something cannot be shared — exactly why.25- Keep the paper and the artifact **consistent**: a bytes-written or latency number in the PDF that26 no script in the artifact regenerates is the contradiction reviewers read as carelessness.2728## Availability statement audit2930| Claim in the paper | Weak answer | FAST-ready answer |31|---|---|---|32| "We evaluate on SSDs A, B, C" | "Standard SSDs" | Exact models, capacities, and **firmware versions** in a testbed table |33| "Driven by trace T" | Trace named, not shared | Archived trace (or documented access) + the replay script and its settings |34| "We reduce write amplification" | Estimated in prose | Device-counter (SMART/log) dumps + the script that computes WA from them |35| "Steady-state results" | Unstated preconditioning | The preconditioning/aging protocol as a runnable script |36| "Our system is available" | "Code on request" | Anonymized, buildable code with a README and a small runnable demo |3738"Available on request" is treated as *not available*; convert every such line into a concrete,39anonymized artifact or an explicit, justified exception (e.g. drives under NDA, privacy-limited40production traces).4142## Provenance pinning4344```text45[Devices] model, capacity, interface, FIRMWARE version; host CPU/RAM, kernel, mkfs/mount options46[State] preconditioning/aging protocol, fill level, TRIM/discard; FOB vs. steady-state disclosed47[Traces] archive the replayed trace or document access; ship the replay tool + timing settings48[Workloads] YCSB/filebench/fio job files with exact parameters and seeds49[Compute] run duration and repeats; note thermal/throttling conditions that affect timing50[Counters] how bytes-written / WA / GC were read from the device, so a reader can re-derive them51```5253## Degrees of reproducibility (state the one you achieved)5455- **Turnkey:** one documented command regenerates a table/figure from logged data (best for56 analysis and plots).57- **Scripted-on-hardware:** scripts run end-to-end but require **comparable devices** and time58 (endurance runs, aging, long trace replays); document the hardware and expected runtime.59- **Descriptive:** prose and provenance detailed enough that a reader with the right hardware could60 rebuild the pipeline (acceptable for results that need specific or proprietary devices).6162For FAST, aim turnkey for anything that runs from logged data, and be honest that device-bound63results are **scripted-on-hardware**: state which drives, which firmware, and how long. Promising64turnkey behavior that silently needs a specific SSD is worse than stating the hardware dependency.6566## The hardware-variance caveat (state it plainly)6768Storage numbers are device- and firmware-specific; a reader on a different drive should expect the69*trend*, not the exact factor. Say so, report **per device** rather than one blended number where it70matters, and bound generalization — this is not a weakness to hide but the honest storage posture71reviewers reward.7273## Vignette: a file-system aging study7475Consider a study whose results depend on a fragmented, aged volume. Its reproducibility spine: the76**aging workload as a runnable script** (so the fragmentation state is reconstructable), the exact77**mkfs/mount options and kernel**, the device models and firmware, the measurement scripts that read78from the device and file-system counters, and the analysis notebooks that turn logs into the paper's79figures — plus one honest sentence about which numbers require the specific drives used.8081## Consistency and camera-ready pass8283- Before submission: every scored number traces to the artifact; the availability statement matches84 reality; the artifact is anonymized (no owner strings, hostnames, cluster paths, or personal trace85 URLs).86- Before camera-ready: swap anonymized links for a permanent, licensed, DOI-issuing archive87 (Zenodo/figshare/Software Heritage) and align the statement with the USENIX artifact badges you are88 pursuing (`fast-artifact-evaluation`).8990## Output format9192```text93[Claim inventory] <claim -> evidence location>94[Availability] concrete / vague / missing95[Provenance gaps] <device+firmware / device state / trace archival / counters / seeds>96[Reproducibility level] turnkey / scripted-on-hardware / descriptive, stated honestly97[Hardware dependency] <which results need which drives/firmware, stated?>98[Paper fixes] <must appear in the PDF>99[Artifact fixes] <additions before upload>100```
brycewang-stanford