brycewang-stanford VLDB Experiments
Use this before the evaluation section hardens. At this venue the experiments are the argument: a PVLDB reviewer typically flips from the introduction straight to the plots and decides how skeptically to read everything else.
What the evaluation must establish
Four distinct burdens, each needing its own experiments:
- The problem exists — measure the baseline failing on a credible workload before showing your fix.
- The mechanism causes the gain — ablate your components; a monolithic "our system vs. theirs" plot proves selection, not mechanism.
- The gain survives scale — curves along data size, cluster size, and concurrency, not one configuration chosen after the fact.
- The cost is known — measure where your design loses and say so.
Workload realism ladder
| Rung | Example | Reviewer credit |
|---|---|---|
| Micro-benchmark | single operator, synthetic keys | Mechanism insight only |
| Standard benchmark | TPC-style, YCSB, JCC-H-class | Comparable, but "benchmark-only" is a known flag |
| Benchmark + skew/drift | standard suite with realistic distributions | Solid |
| Trace or production-derived | replayed real workload | Strongest, disclose provenance |
Climb as high as the data you can legally use allows; state the rung honestly.
Baseline fairness protocol
The people who built your baselines review here. For every competitor:
- Latest stable version (or a justified pin), same hardware, same data.
- Tuning effort comparable to what you gave your own system — document the knobs tried in both cases.
- If a competitor is excluded, one honest sentence why (license, no support for the workload) beats silence.
- Include the strong-but-inconvenient baseline: the hand-tuned config, the single-node engine that wins at small scale, the "just add an index" answer. Reviewers propose these in their first pass; preempt them.
Reporting floor
- Throughput and latency, with tails (p95/p99) where users feel tails.
- Repetition counts and spread (see
vldb-reproducibilityfor the variance protocol); medians for skewed metrics. - Axes from zero or clearly marked; log scales labeled; error bands defined in the caption.
- Every number in the abstract traceable to a specific figure or table.
Ablation and sweep matrix
For each design component C1..Cn:
full system vs full-minus-Ci (mechanism attribution)
For each claimed robustness dimension D:
sweep D across its realistic range (skew, size, concurrency, selectivity)
Loss map:
identify >=1 region where a baseline wins; measure it; discuss it
The loss map is deliberately mandatory. A paper whose system wins everywhere in every plot triggers reviewer search behavior — they will find the losing region themselves, without your framing.
Scale honesty
Run the largest experiments your infrastructure permits, then scope claims to what was run. "Designed for larger deployments" is acceptable prose; extrapolated curves presented as measurements are not. If cloud credits capped the study, say so — builders on the panel have lived that constraint.
Output format
[Burden coverage] problem-exists / mechanism / scale / cost — evidence per burden
[Workload rung] <ladder position, justification>
[Baseline fairness] <competitor -> version, tuning parity, exclusions>
[Reporting floor] tails, reps, spread — gaps
[Loss map] present (region, magnitude) / missing (risk)
[Next decisive run] <the one experiment that most changes acceptance odds>