FAST Reproducibility
Use this before submission and again before camera-ready. FAST's storage results live on real
hardware that ages, throttles, and varies part-to-part, so reproducibility here is a distinct
craft from software-only venues: a reader reproducing your work needs to know not just the code but
the device, its firmware, and its state. The goal is that a competent reader with comparable
hardware could rebuild your evidence and reach your conclusions.
Evidence map
- Map each claim and reported number to a verifiable location — a paper section, a table
generated from a logged run, or a script in the artifact.
- For a system, give enough of the design, parameters, mkfs/mount options, and build environment
that a reader could rebuild and run it.
- For measurements, report the device provenance (models, firmware, interface), the host
(CPU, RAM, kernel), and the device state (steady-state/aged, fill, TRIM) — the storage-only
provenance that software artifacts omit.
- Keep the availability statement truthful and specific: what code and traces are shared, where
they will live after acceptance, and — if something cannot be shared — exactly why.
- Keep the paper and the artifact consistent: a bytes-written or latency number in the PDF that
no script in the artifact regenerates is the contradiction reviewers read as carelessness.
Availability statement audit
| Claim in the paper |
Weak answer |
FAST-ready answer |
| "We evaluate on SSDs A, B, C" |
"Standard SSDs" |
Exact models, capacities, and firmware versions in a testbed table |
| "Driven by trace T" |
Trace named, not shared |
Archived trace (or documented access) + the replay script and its settings |
| "We reduce write amplification" |
Estimated in prose |
Device-counter (SMART/log) dumps + the script that computes WA from them |
| "Steady-state results" |
Unstated preconditioning |
The preconditioning/aging protocol as a runnable script |
| "Our system is available" |
"Code on request" |
Anonymized, buildable code with a README and a small runnable demo |
"Available on request" is treated as not available; convert every such line into a concrete,
anonymized artifact or an explicit, justified exception (e.g. drives under NDA, privacy-limited
production traces).
Provenance pinning
[Devices] model, capacity, interface, FIRMWARE version; host CPU/RAM, kernel, mkfs/mount options
[State] preconditioning/aging protocol, fill level, TRIM/discard; FOB vs. steady-state disclosed
[Traces] archive the replayed trace or document access; ship the replay tool + timing settings
[Workloads] YCSB/filebench/fio job files with exact parameters and seeds
[Compute] run duration and repeats; note thermal/throttling conditions that affect timing
[Counters] how bytes-written / WA / GC were read from the device, so a reader can re-derive them
Degrees of reproducibility (state the one you achieved)
- Turnkey: one documented command regenerates a table/figure from logged data (best for
analysis and plots).
- Scripted-on-hardware: scripts run end-to-end but require comparable devices and time
(endurance runs, aging, long trace replays); document the hardware and expected runtime.
- Descriptive: prose and provenance detailed enough that a reader with the right hardware could
rebuild the pipeline (acceptable for results that need specific or proprietary devices).
For FAST, aim turnkey for anything that runs from logged data, and be honest that device-bound
results are scripted-on-hardware: state which drives, which firmware, and how long. Promising
turnkey behavior that silently needs a specific SSD is worse than stating the hardware dependency.
The hardware-variance caveat (state it plainly)
Storage numbers are device- and firmware-specific; a reader on a different drive should expect the
trend, not the exact factor. Say so, report per device rather than one blended number where it
matters, and bound generalization — this is not a weakness to hide but the honest storage posture
reviewers reward.
Vignette: a file-system aging study
Consider a study whose results depend on a fragmented, aged volume. Its reproducibility spine: the
aging workload as a runnable script (so the fragmentation state is reconstructable), the exact
mkfs/mount options and kernel, the device models and firmware, the measurement scripts that read
from the device and file-system counters, and the analysis notebooks that turn logs into the paper's
figures — plus one honest sentence about which numbers require the specific drives used.
Consistency and camera-ready pass
- Before submission: every scored number traces to the artifact; the availability statement matches
reality; the artifact is anonymized (no owner strings, hostnames, cluster paths, or personal trace
URLs).
- Before camera-ready: swap anonymized links for a permanent, licensed, DOI-issuing archive
(Zenodo/figshare/Software Heritage) and align the statement with the USENIX artifact badges you are
pursuing (
fast-artifact-evaluation).
Output format
[Claim inventory] <claim -> evidence location>
[Availability] concrete / vague / missing
[Provenance gaps] <device+firmware / device state / trace archival / counters / seeds>
[Reproducibility level] turnkey / scripted-on-hardware / descriptive, stated honestly
[Hardware dependency] <which results need which drives/firmware, stated?>
[Paper fixes] <must appear in the PDF>
[Artifact fixes] <additions before upload>
1---2name: fast-reproducibility3description: Use when strengthening USENIX FAST reproducibility and open-science evidence, covering device and firmware provenance, device-state disclosure, trace availability and replay, claim-to-evidence mapping, honest degrees of reproducibility on hardware that ages and varies, and consistency between what the paper says and what the artifact contains.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/fast-reproducibility6---78# FAST Reproducibility910Use this before submission and again before camera-ready. FAST's storage results live on **real11hardware that ages, throttles, and varies part-to-part**, so reproducibility here is a distinct12craft from software-only venues: a reader reproducing your work needs to know not just the code but13the *device, its firmware, and its state*. The goal is that a competent reader with comparable14hardware could rebuild your evidence and reach your conclusions.1516## Evidence map1718- Map each claim and reported number to a **verifiable location** — a paper section, a table19 generated from a logged run, or a script in the artifact.20- For a system, give enough of the design, parameters, mkfs/mount options, and build environment21 that a reader could rebuild and run it.22- For measurements, report the **device provenance** (models, firmware, interface), the **host**23 (CPU, RAM, kernel), and the **device state** (steady-state/aged, fill, TRIM) — the storage-only24 provenance that software artifacts omit.25- Keep the **availability statement** truthful and specific: what code and traces are shared, where26 they will live after acceptance, and — if something cannot be shared — exactly why.27- Keep the paper and the artifact **consistent**: a bytes-written or latency number in the PDF that28 no script in the artifact regenerates is the contradiction reviewers read as carelessness.2930## Availability statement audit3132| Claim in the paper | Weak answer | FAST-ready answer |33|---|---|---|34| "We evaluate on SSDs A, B, C" | "Standard SSDs" | Exact models, capacities, and **firmware versions** in a testbed table |35| "Driven by trace T" | Trace named, not shared | Archived trace (or documented access) + the replay script and its settings |36| "We reduce write amplification" | Estimated in prose | Device-counter (SMART/log) dumps + the script that computes WA from them |37| "Steady-state results" | Unstated preconditioning | The preconditioning/aging protocol as a runnable script |38| "Our system is available" | "Code on request" | Anonymized, buildable code with a README and a small runnable demo |3940"Available on request" is treated as *not available*; convert every such line into a concrete,41anonymized artifact or an explicit, justified exception (e.g. drives under NDA, privacy-limited42production traces).4344## Provenance pinning4546```text47[Devices] model, capacity, interface, FIRMWARE version; host CPU/RAM, kernel, mkfs/mount options48[State] preconditioning/aging protocol, fill level, TRIM/discard; FOB vs. steady-state disclosed49[Traces] archive the replayed trace or document access; ship the replay tool + timing settings50[Workloads] YCSB/filebench/fio job files with exact parameters and seeds51[Compute] run duration and repeats; note thermal/throttling conditions that affect timing52[Counters] how bytes-written / WA / GC were read from the device, so a reader can re-derive them53```5455## Degrees of reproducibility (state the one you achieved)5657- **Turnkey:** one documented command regenerates a table/figure from logged data (best for58 analysis and plots).59- **Scripted-on-hardware:** scripts run end-to-end but require **comparable devices** and time60 (endurance runs, aging, long trace replays); document the hardware and expected runtime.61- **Descriptive:** prose and provenance detailed enough that a reader with the right hardware could62 rebuild the pipeline (acceptable for results that need specific or proprietary devices).6364For FAST, aim turnkey for anything that runs from logged data, and be honest that device-bound65results are **scripted-on-hardware**: state which drives, which firmware, and how long. Promising66turnkey behavior that silently needs a specific SSD is worse than stating the hardware dependency.6768## The hardware-variance caveat (state it plainly)6970Storage numbers are device- and firmware-specific; a reader on a different drive should expect the71*trend*, not the exact factor. Say so, report **per device** rather than one blended number where it72matters, and bound generalization — this is not a weakness to hide but the honest storage posture73reviewers reward.7475## Vignette: a file-system aging study7677Consider a study whose results depend on a fragmented, aged volume. Its reproducibility spine: the78**aging workload as a runnable script** (so the fragmentation state is reconstructable), the exact79**mkfs/mount options and kernel**, the device models and firmware, the measurement scripts that read80from the device and file-system counters, and the analysis notebooks that turn logs into the paper's81figures — plus one honest sentence about which numbers require the specific drives used.8283## Consistency and camera-ready pass8485- Before submission: every scored number traces to the artifact; the availability statement matches86 reality; the artifact is anonymized (no owner strings, hostnames, cluster paths, or personal trace87 URLs).88- Before camera-ready: swap anonymized links for a permanent, licensed, DOI-issuing archive89 (Zenodo/figshare/Software Heritage) and align the statement with the USENIX artifact badges you are90 pursuing (`fast-artifact-evaluation`).9192## Output format9394```text95[Claim inventory] <claim -> evidence location>96[Availability] concrete / vague / missing97[Provenance gaps] <device+firmware / device state / trace archival / counters / seeds>98[Reproducibility level] turnkey / scripted-on-hardware / descriptive, stated honestly99[Hardware dependency] <which results need which drives/firmware, stated?>100[Paper fixes] <must appear in the PDF>101[Artifact fixes] <additions before upload>102```