thedixitjain EDBT Artifact Evaluation
Use this for the artifact and reproducibility package. EDBT's community values re-runnable database-systems work, and the published record is open access on OpenProceedings, so a clean, archived package strengthens the paper and its permanent citation. Two things to internalize: an artifact is judged by an evaluator actually running it, and the review-time package (possibly anonymized) is not the same deliverable as the permanent, DOI-archived package cited at camera-ready.
待核实: whether the current EDBT cycle runs a formal artifact-evaluation track, offers a reproducibility badge, and makes it optional or required is decided per edition — confirm on the current host-site call. The engineering below applies regardless of whether a badge is on offer.
What evaluators open first (database-systems flavor)
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| A system / operator | The build + one run command | Undocumented deps; only-builds-on-authors'-cluster |
| A scalability result | The harness that sweeps sizes/nodes | Numbers with no script; hard-coded to one cluster |
| An Experiments & Analysis study | The comparison harness that regenerates every table | Only the authors' system tuned; missing configs |
| A data/workload contribution | The derivation scripts + the derived data | Description shipped, data missing; provenance unpinned |
Assume an evaluator gives your package a bounded time budget on a machine that is not yours. Design for the first ten minutes — a clean build and a small demo run — to succeed.
Packaging plan
[Container] ship a Dockerfile or a pinned environment (build recipe / lockfile); avoid
"install these 40 things and configure the cluster by hand"
[README] one-screen orientation: what it is, how to build, how to run a demo, how to reproduce
each result, expected runtime and outputs, and what needs a big cluster
[Mapping] an explicit table: paper claim -> script -> expected result (table/figure)
[Workloads] the derived workload/query-log itself (or a documented access path), with pinned
dataset versions, not just a query
[Environment] the hardware/cluster spec assumed, and a reduced-scale path a reviewer can actually run
[License] an OSI-approved code license and clear data terms, compatible with the CC-BY-NC-ND record
[Archive] deposit in a DOI-issuing repository (Zenodo, figshare, Software Heritage) for the
permanent, citable record
Review-time package vs. permanent package
- At submission: if the cycle is double-blind (待核实 per cycle), the package must be anonymized — no owner strings, cluster names, lab paths, or identity-revealing links; if single-blind, still scrub stale paths and credentials.
- After acceptance: deposit the de-anonymized, licensed package in a DOI-issuing archive; this is the version the camera-ready cites and the permanent open-access record points to.
Worked vignette: packaging an operator + evaluation
A paper contributes a query-processing operator and a cluster evaluation. To make it re-runnable:
ship a container with the engine and operator pre-built; a run_demo.sh that runs the operator on a
small bundled workload in under a minute; a reproduce/ directory whose scripts regenerate each
table from logged results and, where feasible, from a fresh reduced-scale run; a claim-to-script
mapping in the README; the derived workloads with pinned dataset versions; and an Apache/MIT license.
State honestly which results are turnkey at small scale and which need the full cluster.
Calibration
- The full-cluster experiments may not be turnkey on an evaluator's machine — provide a reduced-scale path and say clearly what it does and does not reproduce.
- For an Experiments & Analysis paper the artifact stakes are highest: the whole comparison should regenerate, with every system's configuration documented, because the study is the contribution.
- Confirm the current cycle's artifact/reproducibility process, any badge, and its anonymity rules on the host-site call (待核实).
Output format
[Package role] review-time (anonymized?) / permanent DOI-archived
[Contents] <system/build / workloads / harness / provenance / license>
[Ten-minute test] does build + demo succeed on a clean machine? yes/no
[Claim mapping] <claim -> script -> expected result present? yes/no>
[Scale honesty] <what is turnkey small-scale vs. needs a cluster>
[Archive] DOI-issuing repository + compatible license? yes/no
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → EDBT-Skills/skills/edbt-artifact-evaluation/SKILL.md