FAST Experiments
Use this before submission when the storage evaluation is not yet locked. FAST reviewers are storage
people; the evaluation is where a good idea is won or lost, and the questions are storage-specific.
The organizing principle is measure the storage cost you claim to change, on real hardware in a
realistic state — not a throughput bar on a fresh drive.
Evaluation audit
- Match the metric to the storage claim. A claim about endurance needs bytes-written / P/E
cycles, not throughput; a claim about responsiveness needs tail latency (p99/p99.9), not the
mean; a claim about space needs measured on-media footprint; a claim about durability needs a
crash-consistency test. The wrong metric is the most common FAST reject.
- Use real devices, and name them. Model, capacity, interface (SATA/SAS/NVMe), and — critically
— firmware version, plus host, kernel, and filesystem/mkfs options. Nominally identical drives
differ part-to-part and across firmware; a result without the device table is not auditable.
- Control device state. SSDs must be preconditioned to steady state (fresh-out-of-box
numbers flatter every design); report fill level and TRIM/discard state. For file systems, report
whether the volume was aged/fragmented or empty — aging can dominate the result.
- Drive with credible workloads and traces. Standard generators (fio, filebench, YCSB for
KV/DB) and archived traces (SNIA IOTTA block/object traces, production-derived traces) beat an
ad-hoc microbenchmark. Ship the job files and replay scripts.
- Choose fair baselines, including the strongest prior system and a reasonable default, tuned
with a documented, equal budget. An untuned or default-config baseline is a scored weakness.
- Test the invariant your optimization risks. If a change defers writes, batches, or reorders,
show crash consistency still holds (record-and-replay / fault injection), not just that it is
faster.
- Design threats in, not on: know before you run which confounds (device variance, thermal
throttling, cache effects, contamination of a trace) will bite, and instrument to bound them.
Claim-to-evidence design table
| Storage claim |
Matching evidence |
Reject pattern avoided |
| "Cuts write amplification / extends endurance" |
Bytes-written from device counters (SMART/logs) at steady state; projected P/E budget |
"Estimated WA on a fresh drive; no device counters" |
| "Lower/steadier latency" |
Full latency distribution incl. p99/p99.9 under load |
"Reports mean latency only" |
| "Scales to real capacities/workloads" |
Real-sized datasets and standard traces on real devices |
"Tiny dataset on a simulator" |
| "Preserves crash consistency" |
Fault-injection / block-level record-and-replay recovery test |
"Claims consistency, never crash-tests it" |
| "Faster than system X" |
X tuned with equal, documented budget; same hardware and state |
"Default-config or older-hardware baseline" |
| "Reliability finding generalizes" |
Population, models, and duration stated; external validity bounded |
"One model, one datacenter, claimed universal" |
Device-state and measurement floor
[Devices] list model, capacity, interface, firmware; host, kernel, mkfs/mount options
[Steady state] precondition SSDs to steady state; state fill level and TRIM; disclose FOB vs. aged
[Warmup] discard cold-cache warmup unless the cold path IS the claim; state cache/DRAM sizes
[Repeats] multiple runs; report variance/CIs; note thermal or throttling effects
[Counters] read WA, bytes-written, GC activity from device logs where available, not estimates
[Trace replay] replay archived traces with a documented tool; state timing fidelity (open vs. closed loop)
Crash-consistency and durability testing
Durability claims are load-bearing at FAST and are frequently under-tested:
[Model] state the failure model (power loss, kernel panic, fsync semantics)
[Injection] use block-level record-and-replay or a fault injector to cut writes at many points
[Check] verify the post-recovery state satisfies the invariant (no torn/lost committed data)
[Coverage] report how many crash points / orderings were tested, not a single anecdote
Provenance floor for traces and field studies
- Archive the replayed trace (or document access) and the replay scripts; a trace named but not
shipped cannot be reproduced.
- For field/reliability studies, state the population, drive models, technology (SLC/MLC/TLC/QLC),
duration, and how failures/replacements were defined and counted.
- Report how outliers, warmup, and defective units were handled — silent inclusion or exclusion
skews every downstream number.
Vignette: evaluating a compaction change for a KV store
Suppose the paper claims an endurance-aware compaction scheduler cuts bytes written. The matching
plan: run on named SSDs at steady state with firmware recorded; drive with YCSB plus an
archived production trace; measure bytes-written from the device's own counters, not the LSM's
estimate; report read-latency distributions incl. p99.9 to prove the trade is bounded; compare
against the tuned stock compactor with an equal budget; and run a crash-consistency
record-and-replay test to confirm deferring compactions did not weaken durability — every number
traceable to a logged run in the artifact.
Output format
[Evaluation readiness] strong / adequate / weak
[Claim -> metric map] <claim: device/metric/statistic>
[Device reality] <models + firmware + state (steady/aged/fill/TRIM) stated? yes/no>
[Baseline fairness] <baseline -> tuned? equal budget? same hardware/state?>
[Durability check] <crash-consistency / fault-injection test present? yes/no>
[Threats-by-design] <device variance / warmup / trace contamination -> instrumentation>
[Decision-critical next run] <one experiment to add>
Source: brycewang-stanford/Awesome-Journal-Skills → FAST-Skills/skills/fast-experiments/SKILL.md
1---2name: fast-experiments3description: Use when designing or auditing a USENIX FAST storage evaluation, covering real devices and firmware, device-state control (aging, preconditioning, fill, TRIM), standard workloads and traces (SNIA IOTTA, YCSB, filebench, fio), write amplification, tail latency, endurance and wear, crash-consistency testing, fair baselines, and matching the metric to the shape of each storage claim.4---5
6
7# FAST Experiments
8
9Use this before submission when the storage evaluation is not yet locked. FAST reviewers are storage
10people; the evaluation is where a good idea is won or lost, and the questions are storage-specific.
11The organizing principle is **measure the storage cost you claim to change, on real hardware in a
12realistic state** — not a throughput bar on a fresh drive.
13
14## Evaluation audit
15
16- **Match the metric to the storage claim.** A claim about *endurance* needs bytes-written / P/E
17 cycles, not throughput; a claim about *responsiveness* needs **tail latency** (p99/p99.9), not the
18 mean; a claim about *space* needs measured on-media footprint; a claim about *durability* needs a
19 crash-consistency test. The wrong metric is the most common FAST reject.
20- **Use real devices, and name them.** Model, capacity, interface (SATA/SAS/NVMe), and — critically
21 — **firmware version**, plus host, kernel, and filesystem/mkfs options. Nominally identical drives
22 differ part-to-part and across firmware; a result without the device table is not auditable.
23- **Control device state.** SSDs must be **preconditioned to steady state** (fresh-out-of-box
24 numbers flatter every design); report fill level and TRIM/discard state. For file systems, report
25 whether the volume was **aged/fragmented** or empty — aging can dominate the result.
26- **Drive with credible workloads and traces.** Standard generators (fio, filebench, YCSB for
27 KV/DB) and archived traces (SNIA IOTTA block/object traces, production-derived traces) beat an
28 ad-hoc microbenchmark. Ship the job files and replay scripts.
29- **Choose fair baselines,** including the strongest prior system and a reasonable default, tuned
30 with a documented, equal budget. An untuned or default-config baseline is a scored weakness.
31- **Test the invariant your optimization risks.** If a change defers writes, batches, or reorders,
32 show crash consistency still holds (record-and-replay / fault injection), not just that it is
33 faster.
34- **Design threats in, not on:** know before you run which confounds (device variance, thermal
35 throttling, cache effects, contamination of a trace) will bite, and instrument to bound them.
36
37## Claim-to-evidence design table
38
39| Storage claim | Matching evidence | Reject pattern avoided |
40|---|---|---|
41| "Cuts write amplification / extends endurance" | Bytes-written from device counters (SMART/logs) at steady state; projected P/E budget | "Estimated WA on a fresh drive; no device counters" |
42| "Lower/steadier latency" | Full latency distribution incl. p99/p99.9 under load | "Reports mean latency only" |
43| "Scales to real capacities/workloads" | Real-sized datasets and standard traces on real devices | "Tiny dataset on a simulator" |
44| "Preserves crash consistency" | Fault-injection / block-level record-and-replay recovery test | "Claims consistency, never crash-tests it" |
45| "Faster than system X" | X tuned with equal, documented budget; same hardware and state | "Default-config or older-hardware baseline" |
46| "Reliability finding generalizes" | Population, models, and duration stated; external validity bounded | "One model, one datacenter, claimed universal" |
47
48## Device-state and measurement floor
49
50```text
51[Devices] list model, capacity, interface, firmware; host, kernel, mkfs/mount options
52[Steady state] precondition SSDs to steady state; state fill level and TRIM; disclose FOB vs. aged
53[Warmup] discard cold-cache warmup unless the cold path IS the claim; state cache/DRAM sizes
54[Repeats] multiple runs; report variance/CIs; note thermal or throttling effects
55[Counters] read WA, bytes-written, GC activity from device logs where available, not estimates
56[Trace replay] replay archived traces with a documented tool; state timing fidelity (open vs. closed loop)
57```
58
59## Crash-consistency and durability testing
60
61Durability claims are load-bearing at FAST and are frequently under-tested:
62
63```text
64[Model] state the failure model (power loss, kernel panic, fsync semantics)
65[Injection] use block-level record-and-replay or a fault injector to cut writes at many points
66[Check] verify the post-recovery state satisfies the invariant (no torn/lost committed data)
67[Coverage] report how many crash points / orderings were tested, not a single anecdote
68```
69
70## Provenance floor for traces and field studies
71
72- Archive the *replayed* trace (or document access) and the replay scripts; a trace named but not
73 shipped cannot be reproduced.
74- For field/reliability studies, state the population, drive models, technology (SLC/MLC/TLC/QLC),
75 duration, and how failures/replacements were defined and counted.
76- Report how outliers, warmup, and defective units were handled — silent inclusion or exclusion
77 skews every downstream number.
78
79## Vignette: evaluating a compaction change for a KV store
80
81Suppose the paper claims an endurance-aware compaction scheduler cuts bytes written. The matching
82plan: run on **named SSDs at steady state** with firmware recorded; drive with **YCSB plus an
83archived production trace**; measure **bytes-written from the device's own counters**, not the LSM's
84estimate; report **read-latency distributions incl. p99.9** to prove the trade is bounded; compare
85against the **tuned** stock compactor with an equal budget; and run a **crash-consistency
86record-and-replay** test to confirm deferring compactions did not weaken durability — every number
87traceable to a logged run in the artifact.
88
89## Output format
90
91```text
92[Evaluation readiness] strong / adequate / weak
93[Claim -> metric map] <claim: device/metric/statistic>
94[Device reality] <models + firmware + state (steady/aged/fill/TRIM) stated? yes/no>
95[Baseline fairness] <baseline -> tuned? equal budget? same hardware/state?>
96[Durability check] <crash-consistency / fault-injection test present? yes/no>
97[Threats-by-design] <device variance / warmup / trace contamination -> instrumentation>
98[Decision-critical next run] <one experiment to add>
99```
100
101---
102
103**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `FAST-Skills/skills/fast-experiments/SKILL.md`
thedixitjain