brycewang-stanford SOSP Reproducibility
Use this while experiments are being designed and run — months before the SOSP
deadline, not during artifact evaluation. At SOSP the artifact process happens after
acceptance (see sosp-artifact-evaluation), which means reproducibility discipline has
a different job pre-submission: it protects you. Reviewers probe numbers during a
three-month review cycle and a response phase in which new experiments are forbidden;
the only defensible paper is one whose every number can be traced to a logged,
re-runnable measurement of a pinned system.
The OS-research twist: your system changes the platform
An operating-systems artifact often is the environment — a modified kernel, a new scheduler, an interposed I/O path. That collapses the usual app/platform separation and creates specific hazards:
- A rebased kernel patch series can silently change baseline behavior; record the exact base commit and the full patch stack for every run, including baseline runs.
- Firmware, microcode, and mitigations (for example, speculative-execution mitigations) can dominate syscall-heavy microbenchmarks; log them and hold them constant across system and baseline.
- Frequency scaling, turbo states, and NUMA placement move tail latencies by tens of percent; fix the policy, and record it rather than assuming defaults.
Capture the environment mechanically
Hand-written "Experimental Setup" sections drift from reality. Generate the facts:
#!/usr/bin/env bash
# capture-env.sh — run on every experiment node, archive with the run's results
{
uname -a; cat /etc/os-release | head -2
cat /proc/cmdline # mitigations, isolcpus, hugepages
lscpu | grep -E 'Model name|Socket|NUMA|MHz'
free -h | head -2; lsblk -d -o NAME,MODEL,ROTA,SIZE
ip -br link; ethtool eth0 2>/dev/null | grep -E 'Speed|Duplex'
git -C "$REPO" rev-parse HEAD; git -C "$REPO" status --porcelain
} > "env-$(hostname)-$(date +%Y%m%dT%H%M%S).txt"
Archive one such file per node per experiment batch, next to the raw results. When a reviewer asks eleven weeks later whether the baseline ran with the same mitigations, the answer is a file, not a memory.
One command per figure
The standard that survives review pressure: every figure and table in the paper is produced by a script that reads only archived raw logs. No spreadsheet steps, no hand-transcribed numbers. This is also what makes the response phase survivable — you can re-check any reviewer-doubted number against raw data without re-running anything, which is the only kind of "checking" the response rules allow.
| Discipline | Pre-submission payoff | Post-acceptance payoff |
|---|---|---|
| Raw logs archived per run, immutable | Response-phase answers under the no-new-data rule | AE claims map writes itself |
| Figure scripts read logs only | No PDF/data divergence between drafts | Evaluators regenerate your plots |
| Environment capture per node per batch | Detects config drift between system and baseline runs | hardware/ directory is done |
| Run manifest (who, when, which commit, which config) | Attribution when a number looks off | Provenance for the archival artifact |
Variance is a first-class result
Systems effects live in distributions. Reproducibility at SOSP includes making the noise floor explicit:
- Report repetitions and the spread (percentiles or CIs), especially for tail-latency claims — a p99 from one run of one trial is folklore, not a measurement.
- Distinguish sources: run-to-run jitter, node-to-node hardware variation, and time-of-day effects on shared testbeds. If experiments ran on a shared cluster, say so and quantify what that cost in variance.
- Decide the warm-up policy (discard first N iterations? cold-start included?) once, document it, and apply it uniformly; asymmetric warm-up between system and baseline is a classic self-inflicted review wound.
Traces and workloads you cannot publish
Production traces make SOSP evaluations compelling and reproductions hard. The honest pattern: characterize the private trace in the paper (size, arrival statistics, skew, whatever drives the result), release a synthetic generator matched to those statistics, and show at least one headline experiment where synthetic and real traces agree in trend. Flag in the paper which results are re-runnable and which are documented-only — the same tiering the artifact evaluation will ask for later.
Output format
[Repro posture] traceable / partially / folklore
[Environment] capture automated? mitigations+kernel pinned for baseline too?
[Figure pipeline] all one-command? exceptions: <list>
[Variance] repetitions, spread reported, warm-up policy uniform?
[Private data] trace characterization + generator plan
[Gap list] <ordered fixes before the freeze>