Sigmod Artifact Evaluation

SIGMOD Artifact Evaluation

SIGMOD runs a named, long-standing artifact program: the Availability & Reproducibility Initiative (ARI) at reproducibility.sigmod.org. It is optional, happens after acceptance, and is decoupled from the accept/ reject decision — but its badges are embedded into the paper's PDF in the ACM Digital Library, so the payoff is permanent and public. Recent editions registered artifacts through a dedicated HotCRP site (e.g., sigmod25ari.hotcrp.com); confirm the current edition's site and dates before promising a timeline.

The badge ladder

Aim explicitly at one rung. A package engineered for Results Reproduced looks different from one that merely publishes source: it names which figures and tables constitute the "key results" and drives each one end to end.

What ARI evaluators grade

The initiative's stated criteria are coverage (how much of the paper the artifact backs), ease of reproducibility (how little effort a rerun takes), flexibility (can parameters, workloads, and datasets be varied), and portability (does it run beyond the authors' exact machine). Database artifacts fail these in predictable ways:

• Coverage: the artifact rebuilds microbenchmarks but not the headline end-to-end comparison against the competing engine.
• Ease: a 40-step README where a driver script should be.
• Flexibility: scale factors and thread counts hard-coded into binaries.
• Portability: kernel-version, NUMA-layout, or GPU assumptions that only hold on the lab machine; no container or VM escape hatch.

Hardware honesty

Data-management experiments often need big machines. ARI evaluations have historically accommodated this via cloud credits or author-provided access in some editions — but the current edition's policy must be checked, not assumed. Regardless of policy:

• State minimum and recommended hardware in the README's first screen.
• Provide a reduced-scale mode (smaller scale factor, fewer threads) that preserves every qualitative trend, and say which absolute numbers will differ from the paper.
• Wall-clock estimates per experiment; evaluators budget time, and an unannounced 30-hour run is how evaluations stall.

Packaging pattern that passes

artifact/
  README.md          # claims map: Fig/Table -> script -> expected output
  LICENSE
  Dockerfile         # or VM image link; pinned OS + dependency versions
  data/fetch.sh      # pulls public datasets by checksum; sizes stated
  run_all.sh         # full reproduction, prints per-step ETA
  run_small.sh       # laptop-scale variant, trends preserved
  experiments/
    fig7_throughput/ # one directory per paper artifact, self-contained
    tab3_latency/
  plot/              # regenerates the exact PDF figures from raw logs

The claims map is the single highest-leverage file: a table from paper artifact to command to expected output, with tolerances ("within 10% on different hardware; ordering of systems preserved").

Incentives beyond the badge

SIGMOD confers a Best Artifact award through the initiative, and reproducibility reports from past editions are themselves published in the ACM DL — meaning strong artifacts earn citable recognition. For an engine or index paper, the artifact also becomes the de facto baseline implementation future papers must compare against, which compounds citations for years.

Questions evaluators ask that READMEs rarely answer

• Which exact figure numbers count as the paper's key results, and where is that stated?
• What should I see on screen when a run succeeds — and what does a known benign warning look like, so I don't abort a healthy run?
• How much disk does the full dataset expand to after decompression?
• Can steps be resumed after a failure, or does every retry start from data fetch?
• Which numbers are hardware-sensitive, and how much drift is acceptable before I should suspect a real problem?
• Who do I contact (anonymity no longer applies) if a step fails, and how fast do authors respond during the evaluation window?

Answer all six in the README's first two screens and the evaluation's most common failure mode — silent stall and abandonment — mostly disappears.

Sequencing with the paper lifecycle

1. During review: keep the anonymized package coherent (see sigmod-supplementary); it becomes the ARI seed.
2. At camera-ready: publish and tag the release matching published numbers (see sigmod-camera-ready).
3. At ARI registration: submit the tagged version via the edition's HotCRP, then leave it frozen — evaluate-time pushes create version skew.
4. After badging: keep the archive stable; the badge points at the DL record forever.

Output format

[Target badge] Available / Evaluated / Results Reproduced
[Claims map] Fig/Table -> script coverage, gaps listed
[Criteria audit] coverage / ease / flexibility / portability findings
[Hardware plan] full-scale needs, reduced-scale mode, runtimes
[Registration] ARI edition site + dates confirmed or 待核实
[Fix queue] ordered work before artifact submission

Source: brycewang-stanford/Awesome-Journal-Skills → SIGMOD-Skills/skills/sigmod-artifact-evaluation/SKILL.md