brycewang-stanford VLDB Reproducibility
Use this while experiments are still running — reproducibility at a systems venue is an experimental-design property, not a packaging step. The question a PVLDB reviewer silently asks of every performance figure: could a competent lab, given this paper alone, land within noise of these curves?
The disclosure floor
Every performance claim needs its context recoverable from the paper (or its cited artifact):
- Hardware: CPU model and count, memory, storage class and interface, network fabric, node count. "A commodity server" reproduces nothing.
- Software: OS, kernel where it matters (I/O experiments), compiler and flags, and the exact versions and configurations of every system measured — including yours.
- Data: source, size, skew characteristics; for generated data, the generator, its parameters, and its seed.
- Workload: query mix, arrival pattern, client counts, warm-up protocol, and run duration.
- Measurement: what was timed, from where, and what was excluded.
Variance is a systems problem, not a seed problem
ML papers randomize over seeds; systems papers fight nondeterminism from caches, compaction timing, JIT warm-up, thermal throttling, and noisy neighbors. The floor:
| Practice | Rule of thumb |
|---|---|
| Repetitions | ≥3-5 runs per point; state the count |
| Reported statistic | Median or mean — say which; show spread when curves are close |
| Cache state | Declare warm or cold, and how you got there |
| Cloud runs | Same instance placement across systems; note the epoch |
| Background work | Disable or document (compaction, checkpoints, GC) |
A speedup smaller than the run-to-run spread is not a result; either tighten the measurement or drop the claim.
Competitor fairness ledger
Reviewers here often built the systems you compare against. For each baseline record: version or commit, configuration changes from defaults, tuning effort spent, and any feature disabled — then disclose that ledger in the paper. An untuned competitor found by its author on the program committee is a one-review rejection.
Traceability from figure to raw data
paper figure N
<- plots/make_fig_N.py
<- results/expN/*.csv (raw, one file per run)
<- run.sh expN --config configs/expN.yaml
<- git tag paper-vN + Dockerfile digest
Build this chain during the project, not after acceptance. It is what makes the revision window survivable — a reviewer-requested variation becomes a config edit instead of archaeology — and it is exactly what the pVLDB Reproducibility Committee will walk if you enter the evaluation.
The repro-honesty paragraph
State in the paper what is not reproducible and why: proprietary traces, production-only scale, licensed competitors. PVLDB's culture (availability badges, mandatory EA&B evaluation) rewards declared limits and punishes discovered ones. One honest paragraph outperforms a broken promise of full reproducibility.
Output format
[Disclosure floor] met / gaps (hardware/software/data/workload/measurement)
[Variance handling] reps, statistic, spread shown — weak points
[Competitor ledger] complete / untuned or unpinned baselines listed
[Trace chain] figure->script->raw->tag intact / broken links
[Declared limits] <what is stated as non-reproducible and why>
[Highest-risk claim] <claim whose evidence would not survive a rerun>