brycewang-stanford VLDB Artifact Evaluation
Use this once a PVLDB paper is accepted (or when an EA&B submission is being planned, since participation is not optional there). The pVLDB Reproducibility Evaluation — run jointly with SIGMOD's effort since the 2018 push — has committee members rebuild your experiment from your package. Two distinct prizes exist: the ACM availability badge for sharing, and the Reproducible outcome (with a Best Reproducible Paper Award) for surviving an independent rerun.
Who must play, who should
| Situation | Obligation |
|---|---|
| EA&B paper | Required: release all data and software, submit to evaluation |
| Regular research paper | Optional but strongly encouraged; badge on offer |
| Industrial paper with proprietary core | Availability of what can be shared; document the rest |
| Vision paper | Rarely applicable |
The four surfaces evaluators rebuild
The committee's published expectations decompose an artifact into four layers. Package each one explicitly:
- Prototype — source code, build environment, configuration. A container image plus the Dockerfile that produced it is the community's default.
- Input data — the datasets themselves, or deterministic generators with pinned seeds and a size knob, plus download scripts for public corpora.
- Workload — the exact queries, client configuration, thread counts, and run durations behind every experiment, not a representative sample.
- Analysis — scripts that transform raw measurements into each numbered figure and table in the PDF. This layer is the one authors most often skip and evaluators most often need.
Design for a stranger's machine
- Assume the evaluator has no access to your cluster. Provide a scaled-down mode that demonstrates every claim's shape on one commodity machine, and document how the full-scale numbers were obtained.
- Pin everything: base images, package versions, competitor-system commits. "Latest" is where reruns go to die.
- Emit expected outputs and tolerances. A rerun that produces a plot is only useful if the evaluator can tell whether the plot is right — state which qualitative relationships must hold even when absolute numbers shift with hardware.
- Time-box honestly: state wall-clock cost per experiment so the committee can schedule, and mark the one experiment that best represents the paper if resources run short.
Minimal package skeleton
artifact/
README.md # claims map: figure/table -> command -> expected shape
Dockerfile # or image reference + build recipe
data/get_data.sh # fetch or generate, seeded
workloads/ # exact configs per experiment
run_one.sh <exp> # single experiment, scaled-down default
run_full.sh # full-scale protocol, hardware stated
plots/make_all.sh # raw results -> paper figures
Award positioning
Winning packages read like engineering products: one command to a first
result, claims mapped to figures, failures anticipated. If the evaluation
report will say "worked on the first try," you are in contention; if it says
"worked after correspondence with the authors," you got the badge and lost the
award. Current-cycle evaluation logistics and criteria wording: 待核实 on
vldb.org/pvldb/reproducibility before packaging.
Output format
[Track] EA&B-mandatory / voluntary / availability-only
[Surface coverage] prototype / data / workload / analysis — gaps listed
[Stranger test] scaled-down mode exists / cluster-only (risk)
[Pinning] images, versions, competitor commits — unpinned items
[First-command experience] <what happens>
[Fixes before submission to the committee] <ordered>