SoCC Experiments
Use this before submission when the evaluation is not yet locked. SoCC reviewers come from both
SIGMOD and SIGOPS, and the evaluation is where a good cloud idea is won or lost. The organizing
principle is measured evidence proportional to the claim — the study must test the thing the
paper asserts, on a system and workloads a skeptic from either community would accept, with tail
latency and cost reported, not just the mean.
Evaluation audit
- Prefer a real or realistic deployment over simulation. A cloud claim about a system needs the
system running on a testbed or cluster; simulation alone invites the "does it hold on real
hardware?" reject. Where full scale is impossible, run a faithful scaled deployment and say so.
- Use production or representative workloads/traces, with stated provenance and selection, and
release the replay harness. Synthetic-only microbenchmarks invite the "does this reflect real
load?" objection.
- Report tail and cost, not just averages. p95/p99 (p99.9 where it bites), the latency
distribution, and a concrete cost model (instance-seconds or $) are the outcomes operators are
billed against.
- Choose fair baselines, including the strongest prior system and a simple-but-reasonable
alternative, each tuned with a documented, equal budget. An untuned baseline is a scored weakness.
- Show scale and multi-tenancy behavior: how the result holds as nodes, tenants, or load grow,
and where the bottleneck appears — SoCC papers live at scale.
- Make the measurement pipeline reproducible (see the code block) so the evaluation reproduces
rather than re-samples.
- Design limits in, not on: know before you run which provider, region, and regime limits the
study will have, and instrument to bound them.
Claim-to-evidence design table
| Cloud claim |
Matching evidence |
Reject pattern avoided |
| "System raises throughput at scale" |
Throughput across node counts on a real testbed vs. tuned baseline |
"Simulated / small-scale only" |
| "Holds tail latency under bursts" |
p99 distribution under bursty replayed load |
"Only mean latency reported" |
| "Cuts cost" |
Instance-seconds or $ under a stated pricing model |
"Cost asserted, never measured" |
| "Fair under multi-tenancy" |
Per-tenant SLO attainment with contending workloads |
"Single-tenant microbenchmark" |
| "The new mechanism adds the value" |
Ablation isolating the mechanism vs. a simpler policy |
"Mechanism's marginal value never isolated" |
Tail, cost, and scale floor
[Tail] report percentiles and the distribution, not just the mean; state the SLO target
[Cost] define the pricing model; report the cost metric so a reader can recompute the saving
[Scale] sweep nodes/tenants/load; identify the bottleneck where the result degrades
[Runs] report the number of runs and the variance for any measured metric
[Compute] state the testbed actually used (nodes, instance types, kernel), not vague feasibility
Workload and trace provenance floor
- Pin commit SHAs and record trace extraction dates; archive the replayed trace or a faithful
generator, not just a pointer or a query.
- State inclusion/exclusion criteria and the resulting workload, with the filtering/replay scripts
in the artifact.
- Report how the trace maps onto the testbed (scaling factors, time compression) so a reader can
judge realism.
Vignette: evaluating a storage QoS mechanism
Suppose the paper claims a scheduler enforces latency SLOs on shared storage better than the prior
system. The matching plan: run both on a real storage testbed with a mix of latency-sensitive and
batch tenants replayed from a representative workload; tune both with an equal, documented budget;
report per-tenant p99 attainment and aggregate throughput with variance; show behavior as tenants
and load scale and name the bottleneck; report the provisioning cost of each; and state
external-validity limits (storage backend, workload mix) — every number traceable to a logged run in
the artifact.
Statistical and measurement reporting floor
- Latency distributions and tail percentiles for every comparison, not point estimates.
- Number of runs and the source of variance for any measured metric.
- A stated cost model behind any economic claim; the compute actually consumed.
Output format
[Evaluation readiness] strong / adequate / weak
[Claim -> evidence map] <claim: testbed/workload/metric incl. tail+cost>
[Deployment realism] <real testbed / scaled deployment / simulation-only>
[Baseline fairness] <baseline -> tuned? equal budget? documented?>
[Tail+cost+scale] <percentiles + cost model + scaling behavior reported? yes/no>
[Provenance] <trace SHAs+dates, replay harness, testbed described? yes/no>
[Decision-critical next run] <one experiment or measurement extension>
Source: brycewang-stanford/Awesome-Journal-Skills → SoCC-Skills/skills/socc-experiments/SKILL.md
1---2name: socc-experiments3description: Use when designing or auditing ACM SoCC evaluations, covering real or realistic deployments over simulation, production or representative workloads and traces, tail-latency and cost as first-class metrics, fair tuned baselines, scale and multi-tenancy behavior, reproducible measurement pipelines, and matching evidence to the shape of each cloud claim.4---5
6
7# SoCC Experiments
8
9Use this before submission when the evaluation is not yet locked. SoCC reviewers come from **both
10SIGMOD and SIGOPS**, and the evaluation is where a good cloud idea is won or lost. The organizing
11principle is **measured evidence proportional to the claim** — the study must test the thing the
12paper asserts, on a system and workloads a skeptic from either community would accept, with **tail
13latency and cost** reported, not just the mean.
14
15## Evaluation audit
16
17- **Prefer a real or realistic deployment over simulation.** A cloud claim about a system needs the
18 system running on a testbed or cluster; simulation alone invites the "does it hold on real
19 hardware?" reject. Where full scale is impossible, run a faithful scaled deployment and say so.
20- **Use production or representative workloads/traces,** with stated provenance and selection, and
21 release the replay harness. Synthetic-only microbenchmarks invite the "does this reflect real
22 load?" objection.
23- **Report tail and cost, not just averages.** p95/p99 (p99.9 where it bites), the latency
24 distribution, and a concrete cost model (instance-seconds or $) are the outcomes operators are
25 billed against.
26- **Choose fair baselines,** including the strongest prior system and a simple-but-reasonable
27 alternative, each tuned with a documented, equal budget. An untuned baseline is a scored weakness.
28- **Show scale and multi-tenancy behavior:** how the result holds as nodes, tenants, or load grow,
29 and where the bottleneck appears — SoCC papers live at scale.
30- **Make the measurement pipeline reproducible** (see the code block) so the evaluation reproduces
31 rather than re-samples.
32- **Design limits in, not on:** know before you run which provider, region, and regime limits the
33 study will have, and instrument to bound them.
34
35## Claim-to-evidence design table
36
37| Cloud claim | Matching evidence | Reject pattern avoided |
38|---|---|---|
39| "System raises throughput at scale" | Throughput across node counts on a real testbed vs. tuned baseline | "Simulated / small-scale only" |
40| "Holds tail latency under bursts" | p99 distribution under bursty replayed load | "Only mean latency reported" |
41| "Cuts cost" | Instance-seconds or $ under a stated pricing model | "Cost asserted, never measured" |
42| "Fair under multi-tenancy" | Per-tenant SLO attainment with contending workloads | "Single-tenant microbenchmark" |
43| "The new mechanism adds the value" | Ablation isolating the mechanism vs. a simpler policy | "Mechanism's marginal value never isolated" |
44
45## Tail, cost, and scale floor
46
47```text
48[Tail] report percentiles and the distribution, not just the mean; state the SLO target
49[Cost] define the pricing model; report the cost metric so a reader can recompute the saving
50[Scale] sweep nodes/tenants/load; identify the bottleneck where the result degrades
51[Runs] report the number of runs and the variance for any measured metric
52[Compute] state the testbed actually used (nodes, instance types, kernel), not vague feasibility
53```
54
55## Workload and trace provenance floor
56
57- Pin commit SHAs and record trace extraction dates; archive the *replayed* trace or a faithful
58 generator, not just a pointer or a query.
59- State inclusion/exclusion criteria and the resulting workload, with the filtering/replay scripts
60 in the artifact.
61- Report how the trace maps onto the testbed (scaling factors, time compression) so a reader can
62 judge realism.
63
64## Vignette: evaluating a storage QoS mechanism
65
66Suppose the paper claims a scheduler enforces latency SLOs on shared storage better than the prior
67system. The matching plan: run both on a real storage testbed with a mix of latency-sensitive and
68batch tenants replayed from a representative workload; tune both with an equal, documented budget;
69report per-tenant p99 attainment and aggregate throughput with variance; show behavior as tenants
70and load scale and name the bottleneck; report the provisioning cost of each; and state
71external-validity limits (storage backend, workload mix) — every number traceable to a logged run in
72the artifact.
73
74## Statistical and measurement reporting floor
75
76- Latency **distributions and tail percentiles** for every comparison, not point estimates.
77- Number of runs and the source of variance for any measured metric.
78- A stated cost model behind any economic claim; the compute actually consumed.
79
80## Output format
81
82```text
83[Evaluation readiness] strong / adequate / weak
84[Claim -> evidence map] <claim: testbed/workload/metric incl. tail+cost>
85[Deployment realism] <real testbed / scaled deployment / simulation-only>
86[Baseline fairness] <baseline -> tuned? equal budget? documented?>
87[Tail+cost+scale] <percentiles + cost model + scaling behavior reported? yes/no>
88[Provenance] <trace SHAs+dates, replay harness, testbed described? yes/no>
89[Decision-critical next run] <one experiment or measurement extension>
90```
91
92---
93
94**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `SoCC-Skills/skills/socc-experiments/SKILL.md`
thedixitjain