brycewang-stanford EuroSys Reproducibility
Use this while experiments are still running — reproducibility retrofitted in
deadline week is transcription, not engineering. The venue context: EuroSys
papers live on measured performance claims, the community runs a badge-granting
artifact evaluation (sysartifacts.github.io), and SIGOPS has publicly digested
five years of EuroSys AE lessons (sigops.org blog, 2025; rendered 2026-07-08).
A paper whose numbers cannot be regenerated by its own authors three months
later fails both review-time scrutiny and post-acceptance AE.
The provenance ledger
Keep one machine-readable record per reported number. Minimum fields:
| Field | Why EuroSys reviewers care |
|---|---|
| Commit hash of system under test | "Which version got 2.1x?" is a real AE question |
| Baseline name + version + config | Untuned-baseline suspicion is the venue's default |
| Hardware: CPU/RAM/NIC/storage, topology | Systems results rarely transfer across boxes |
| OS/kernel, key library versions | Kernel changes move I/O and scheduler numbers |
| Workload/trace + generator seed | Trace provenance is checked, not assumed |
| Repetitions, warm-up policy | Distinguishes measurement from anecdote |
| Timestamp + raw-output path | Lets you rebuild any figure from raw logs |
Variance discipline
Single-run numbers are the most common silent reproducibility failure in systems evaluation:
- Repeat every headline measurement enough times to see its spread; report median plus an explicit dispersion measure (stdev, IQR, or min–max), and say in the caption which one it is.
- Isolate noise sources you control: pin frequencies, disable turbo where it distorts comparisons, note co-located load, randomize run order across systems so drift does not favor yours.
- Tail metrics (p99 and beyond) need far more samples than means; state the sample count whenever a tail latency is claimed.
- When a difference is within run-to-run spread, say so — EuroSys reviewers reward calibrated claims over uniform victory narratives.
Automation floor
The practical bar: any figure regenerates from raw data with one command.
# Layout that keeps figures honest
experiments/
fig7_throughput/
run.sh # executes the sweep, writes results/*.csv with metadata header
plot.py # reads results/, emits fig7.pdf — no hand-edited numbers
results/ # raw outputs, never overwritten, one dir per run timestamp
make fig7 # the only path by which fig7.pdf ever changes
If a plot was ever touched manually, its provenance is broken and the AEC will find the discrepancy before you do.
Availability statement, two audiences
- Review time (double-blind): describe what exists — "an anonymized repository containing the system, workload generator, and run scripts accompanies the submission" — without leaking the lab's identity through URLs, paths, or commit authors.
- Camera-ready / AE time: replace with the DOI-backed archive and the badge set being sought. Restricted traces (production data, partner NDAs) need an honest fallback: a synthetic generator calibrated to the trace's published statistics, with the calibration method described.
Restricted evidence, stated honestly
Some EuroSys evidence legitimately cannot ship — production traces under NDA, partner clusters, proprietary workloads. The honest pattern:
- Name the restriction and its scope precisely ("the ingestion trace from operator X cannot be released; its summary statistics are in Table 3").
- Ship a calibrated synthetic substitute and the calibration procedure, so external readers can approximate the regime.
- Keep at least one headline result on fully public inputs; a paper whose every number depends on unreleasable data asks reviewers for faith the venue does not trade in.
- Never let the availability paragraph imply more openness than the AEC will find; the badge process makes overstatement visible in print.
Pre-deadline reproducibility drill
One week before the paper gate, run the drill on a machine that never ran the experiments:
- Clean checkout, environment build from the lockfile alone — record every undocumented step it turns out to need.
- Regenerate two figures end to end: one cheap, one expensive.
- Diff regenerated numbers against the draft's numbers; investigate any drift beyond the reported dispersion.
- Fix the documentation, not just the outcome — the drill's product is the README the AEC will eventually read.
Quick self-test
- Can a new student regenerate Figure 7 from a clean checkout in one command?
- Does every table cell trace to a raw log file with hardware metadata?
- Are repetition counts and dispersion visible for every performance claim?
- Would the availability paragraph survive both anonymity and the AEC?
Output format
[Repro grade] regenerable / scripted-with-gaps / manual
[Ledger coverage] <numbers with full provenance / total reported numbers>
[Variance findings] <single-run claims, missing dispersion, undersampled tails>
[Workload provenance] <traces and generators with version + seed status>
[Availability draft] <review-time text and camera-ready plan>