Eurosys Experiments

EuroSys Experiments

Use this while the evaluation is still designable. EuroSys reviewers treat the evaluation as the paper's testimony about itself: the design section says what should happen, and §Evaluation is cross-examination. Plan it as a set of questions with falsifiable answers, not as a benchmark tour.

The four-layer evidence stack

A submission with only the first layer reads as a demo; the middle layers are what convert "it is faster" into "we understand why it is faster", which is the systems-research standard of proof.

Workload realism, argued not asserted

• Prefer published traces and standard suites with named versions; when using a synthetic generator, calibrate it against a real distribution and say how.
• Match the workload to the claim's regime: a memory-efficiency claim needs memory pressure; a tail-latency claim needs load near saturation, not at 30%.
• Scale honestly: if the pitch is rack-scale, a two-node result needs an explicit extrapolation argument or a scoped-down claim.
• Report the workload's own parameters (skew, read ratio, arrival process) so the experiment is reconstructible without your cluster.

Baseline fairness — the venue's sharpest knife

The reflexive EuroSys reviewer question is "did they tune the baseline?" Answer it before it is asked:

• Run baselines at their documented best configuration for your hardware, and cite where that configuration comes from.
• Include the strongest deployed alternative, not only research prototypes — losing to a well-tuned production system on some axis is survivable; omitting it is not.
• Same hardware, same workload generator, same measurement harness for every system; any asymmetry gets a sentence of justification.
• When your system loses a metric, plot it anyway and explain the tradeoff.

Experiment matrix as an artifact

Freeze the plan in a machine-checkable form before running:

# eval-matrix.yaml — one row per claim the paper will make
- claim: "cuts p99 GET latency ≥40% under skewed load"
  figure: fig8
  workload: {trace: twitter-cache-2020, skew: zipf-0.99, load: 0.85sat}
  systems: [ours@v1.4, baselineA@3.2-tuned, baselineB@1.9-tuned]
  reps: 10
  metrics: [p50, p99, p999, throughput]
  status: pending

The matrix doubles as the reproducibility ledger (eurosys-reproducibility) and later as the artifact's claims map (eurosys-artifact-evaluation).

Result invalidators to design out early

Each of these has sunk otherwise strong EuroSys evaluations; each is cheap to prevent and expensive to discover in a review:

• Measuring through a bottleneck that is not the system under test — a saturated client, a slow log disk, a debug build of a dependency.
• Comparing your warm system against cold baselines (or vice versa) because warm-up policy was never standardized across systems.
• A "scalability" curve whose x-axis grows load and resources together, so nothing about scaling behavior is actually isolated.
• Reporting means over runs that include documented failures — decide the failure-handling policy for measurements before running them.
• Config drift between the end-to-end and decomposition experiments, so the component gains do not sum to anything resembling the headline.

Sizing runs for the claim

• Comparative bar charts: ≥5–10 repetitions with dispersion shown; single-run bars invite a one-line rejection rationale.
• Tail latencies: enough requests that the quoted percentile has support — a p999 from 10k requests is ten samples of noise.
• Ablations: vary one factor per experiment; a config delta of two changes attributes nothing.
• Timeboxing: reserve cluster time for the boundary layer explicitly; it is always the layer teams drop under deadline pressure and the layer reviewers miss loudest.

Reporting floor for the paper

• Hardware and topology for every experiment, once, in a table the reader can find (§Evaluation setup), with per-figure deviations noted.
• Software versions for every system in every comparison, including yours.
• Load points quoted with the metric ("85% of measured peak throughput"), never as bare client counts whose meaning depends on the cluster.
• Repetition count and dispersion type in every caption that shows a comparison.
• A sentence on measurement methodology for anything subtle: how latency is timestamped, where the measurement harness sits, what it costs.

Output format

[Claim -> evidence map] <each paper claim: figure, workload, systems, reps>
[Stack coverage] end-to-end / decomposition / cost / boundary: present or missing
[Baseline fairness] <tuning provenance and deployed-alternative status>
[Realism audit] <trace provenance, load regime, scale honesty>
[Highest-value missing run] <the one experiment to schedule next>

Source: brycewang-stanford/Awesome-Journal-Skills → EuroSys-Skills/skills/eurosys-experiments/SKILL.md