brycewang-stanford NSDI Reproducibility
A networked-systems result is a function of topology, traffic, timing, and code — and three of those four are absent from the PDF unless deliberately recorded. NSDI rewards the discipline institutionally: artifact badges after acceptance and a Community Award for the best paper whose code and/or dataset is public by the final-papers deadline. But the work happens during the experiments, not after the decision email.
The four provenance ledgers
Keep each as a versioned file next to the results it explains:
| Ledger | Contents | Loss mode it prevents |
|---|---|---|
| Topology | node specs, NIC/switch models, link speeds, RTT matrix, kernel + NIC settings | "worked on our cluster," unreproducible knee points |
| Traffic | trace source + collection context, scaling/anonymization transforms, synthetic-generator parameters + seeds | headline numbers tied to data nobody can regenerate |
| Configuration | every config diff from defaults, per system and per baseline | untuned-baseline accusations you cannot rebut |
| Run | per-experiment: commit hash, config snapshot, fault schedule, raw-log location, analysis-script hash | figures that cannot be regenerated at revision time |
The run ledger has a second life NSDI makes unusually likely: a one-shot revision letter can demand new measurements on the same setup, 3-4 months after submission, at a subsequent deadline. Teams with ledgers rerun in days; teams without them rebuild the testbed from memory.
Determinism is different here
Unlike ML reproducibility, networked-systems results are legitimately non-deterministic — background load, timer jitter, and cross-traffic vary. The honest posture is not "identical numbers" but characterized variance:
- Fix what can be fixed: seeds for generators and fault schedules, pinned software versions, isolated or documented-shared testbeds.
- Measure what cannot: repeat headline experiments across days/trace slices; publish the spread with the artifact so an evaluator knows a 7% delta is noise, not failure.
- Timestamp everything; time-of-day and neighbor effects are real explanations, but only if recorded.
What can actually ship?
Decide per data item before the camera-ready crunch, because the answer shapes the paper's claims:
- Public now — code, synthetic generators, testbed configs, analysis scripts. Default to shipping.
- Public after transformation — production-derived traces that can be anonymized/aggregated. Document the transform and what properties it preserves; state in the paper that released traces differ from raw ones.
- Never public — proprietary traces, customer data. Plan the fallback: a synthetic workload with matched characteristics, shipped alongside a fidelity note. Do not let a never-public trace be the only support for a headline claim if a shippable proxy can corroborate it.
Anonymize the artifact itself for review-time supplements (repo owners, hostnames, paths, company strings in configs); de-anonymize only at final-paper time.
Wiring it into the repo
# Layout that keeps paper and artifact from drifting
experiments/
<exp-id>/run.sh # topology + config + fault schedule, self-describing
<exp-id>/provenance.json # commit, trace id, seeds, dates, operator
figures/
Makefile # every paper figure regenerated from raw logs:
# make fig6 -> pulls exp logs, runs analysis, emits PDF
paper/
claims.md # claim -> exp-id -> figure mapping, reviewed at freeze
The claims.md cross-map is the cheapest anti-drift device: at submission freeze,
walk it once; any claim whose exp-id is stale gets rerun or reworded. The shared
smoke-checker
(../../resources/code/README.md) covers package
hygiene but not topology/trace fidelity — those checks are yours.
Blind now, open later
Reproducibility material crosses the double-blind boundary twice, and each crossing has a checklist:
- At submission (if any package is uploaded as auxiliary material): scrub repository owners, commit-author emails, hostnames that embed the institution, and license files naming the organization; keep the scrubbed and true versions as separate branches so de-anonymization is a merge, not a rewrite.
- At final papers: flip to the public, attributed release — DOI'd archive,
real repository, institutional acknowledgment — dated before the final-papers
deadline if the Community Award matters (
nsdi-camera-ready).
Cheap wins ordered by payoff
provenance.jsonper experiment, written by the runner script (an hour to automate).- Figure Makefile with raw-log inputs (removes an entire class of revision pain).
- Baseline config diffs recorded (rebuts the most common reviewer attack).
- Variance characterization for the two headline results.
- Early legal check on trace release — licensing takes longer than deadline gaps.
Signs the discipline is slipping
- A figure in the draft that nobody can regenerate this week.
- "The good run" language — selecting runs by outcome rather than by protocol.
- Baseline numbers copied forward from an earlier paper draft instead of re-measured on the current topology.
- Testbed changes (kernel upgrade, NIC swap) mid-campaign without a ledger entry splitting before/after results.
Any one of these, found at submission freeze, costs days; found by an artifact evaluator or a revision reviewer, it costs the result's credibility.
Output format
[Ledger status] topology / traffic / configuration / run — each present/partial/absent
[Variance] headline results with characterized spread? which lack repeats?
[Shippability] data items -> public / transformable / never (fallback named)
[Drift check] claims.md walked? stale claim list
[Award posture] on track for public code+data by final-papers deadline?
[Next actions] ordered by payoff-per-hour