brycewang-stanford OSDI Experiments
Design the evaluation as the paper's proof obligation. The page constraints referenced here are OSDI '26 rules (12 reviewed pages, no appendices at submission — verified 2026-07-08); the evidence standards are the durable expectations of systems PCs.
Research questions first
Write the evaluation's research questions before running anything, and derive the experiment set from them. Every OSDI evaluation ultimately answers versions of:
- Does the idea work end to end? — the headline comparison on a realistic workload against the strongest baseline.
- Where does the benefit come from? — component breakdown attributing the win to the named design idea rather than to incidental engineering.
- What does it cost? — the overheads the design admits (memory, write amplification, CPU, complexity), measured, not estimated.
- When does it break? — scalability limits, adversarial workloads, failure and recovery behavior.
An evaluation organized as RQ1–RQ4 with one experiment cluster each reads as an argument; a tour of every benchmark you happened to run reads as padding, which the OSDI '26 CFP explicitly invites reviewers to down-rank.
Baselines that fight back
The baseline question decides more OSDI reviews than any other. Standards:
- Compare against the strongest deployed or published system for the problem, at its tuned best — not a strawman configuration. Reviewers who built the baseline will recognize a sandbagged setup instantly.
- If the nearest competitor is unavailable, re-implement from its paper and say so, with the re-implementation validated against published numbers where possible.
- Include the do-nothing baseline where it is honest: sometimes the existing system plus more hardware is the real alternative, and the paper is stronger for pricing it.
- Version, configuration, and tuning of every baseline belong in the paper. Under the no-appendix rule this is body text — budget for it.
Workload realism
| Workload tier | Role in the argument | Trap |
|---|---|---|
| Microbenchmarks | Isolate a mechanism; explain why the end-to-end effect exists | As the only evidence: workshop-grade |
| Standard suites (e.g., YCSB-class) | Comparability with prior papers | Defaults nobody runs in production |
| Trace-driven / production-derived | The claim's load-bearing evidence | Provenance undocumented (see osdi-reproducibility) |
| Adversarial / stress | Answers RQ4 honestly | Omitted, leaving reviewers to imagine worse |
Systems reviewers read workload sections looking for the flattering-choice smell: the one skew setting, working-set size, or thread count where the design shines. Sweep the parameter, show the crossover point, and say where the baseline wins — a visible crossover is credibility, not weakness.
Measurement discipline
- Report distributions, not averages, wherever latency matters: medians and p99s diverge exactly where systems papers live. State run counts and variance.
- Separate warm and cold behavior; state measurement windows and what was discarded.
- Attribute wins: a profile or counter-level breakdown connecting the speedup to the mechanism ("the win disappears when we disable per-tenant ordering") beats any additional benchmark.
- Failure experiments need injected faults with stated injection method and timing, not prose about what recovery "would" do.
Experiment matrix skeleton (freeze ~8 weeks before the December deadline):
RQ | workload (tier + provenance) | baselines (version, tuning) | metric
| scale points | runs x seeds | expected figure/table | status
Freeze the matrix, then let deadline pressure cut rows, never redefine them —
redefinition under pressure is how flattering choices happen.
Fitting evidence into 12 pages
With no appendix at submission, the evaluation must be self-sufficient and compact:
- One figure or table per research question, each with a caption stating its takeaway.
- Cut experiments that do not serve an RQ, even if they took weeks — the accepted-paper
14-page budget (plus appendices) can resurrect them later (
osdi-camera-ready). - Keep the setup paragraph brutal: hardware, topology, versions, workloads, runs — in
one place, once (
osdi-reproducibilityowns the full ledger).
Reporting grid
Match each metric class to its honest presentation before making figures:
| Metric class | Report as | Not as |
|---|---|---|
| Throughput | Curve vs offered load, to saturation | Single peak number |
| Latency | Median + p99 (p999 if claimed), distribution across runs | Mean ± nothing |
| Recovery/failover | Timeline from fault injection, per scale point | "Fast recovery" prose |
| Overhead (the design's cost) | Same rigor as the win, same table | Footnote estimate |
| Scalability | Efficiency vs ideal at each point | "Near-linear" unquantified |
One convention repays its cost: keep the baseline's color/marker identical across
every figure, so the skim (osdi-review-process) reads the comparison correctly
without consulting legends.
Review-time exposure
The evaluation objections you cannot rebut (no response period in 2026) are the predictable ones: weak baseline, unrealistic workload, missing cost measurement, and average-only latency. Audit for exactly these four before submission; each unaddressed one is a review point conceded silently.
Output format
[RQ coverage] RQ1-4 each mapped to experiments? gaps: <list>
[Baseline verdict] strongest opponent present + tuned? <one-line judgment>
[Workload realism] tiers present; flattering-choice risks: <list>
[Cost honesty] design's costs measured? <which, where>
[Tail discipline] distributions + variance reported? <yes/no + fix>
[Page fit] evaluation length vs 12-page budget; cut candidates: <list>