ICDE Artifact Evaluation
Use this for evidence packaging around ICDE. ICDE expects authors to submit supplemental
material and considers its availability in the evaluation, so a strong artifact directly
raises the paper's floor even where no separate badge process runs. Confirm whether the current
edition also runs a post-acceptance reproducibility/badge track (待核实) before promising
evaluators anything.
Artifact plan
- Decide what evidence a builder needs to believe the numbers: the system source, the
workload generators, the baselines' configurations, the datasets or their construction
scripts, seeds, and logs.
- Keep decision-critical evidence in the paper or its figures; the artifact demonstrates
reproducibility, it does not replace the argument.
- Provide a minimal reproduction map: build steps, dependencies, hardware assumptions
(especially the storage device), commands, expected outputs, runtime, and known
nondeterminism sources.
- For restricted or proprietary data, give enough provenance and construction detail for a
credible re-run without violating data-use terms.
- After acceptance, publish the public, licensed, tagged version whose commit produced the
paper's numbers.
What ICDE evidence reviewers open first
| Claim type |
First artifact inspected |
Common failure caught |
| Throughput/latency win |
run_small.sh and the workload generator |
Numbers cannot be regenerated; generator or seeds absent |
| "Mechanism causes the gain" |
The ablation toggle in the code |
The toggle does not exist; gain not isolable in the artifact |
| Baseline comparison |
The baseline's config files |
Baseline was untuned or run with defaults |
| Scale claim |
The scale-factor sweep script |
Only one scale factor is actually runnable |
| Cost/overhead claim |
The instrumentation that measures cost |
Cost is asserted but not measured anywhere |
Because ICDE reviewers are builders, they will re-run a small benchmark far sooner than they
will provision a cluster — make run_small.sh reproduce the headline crossover on one machine
in minutes before polishing anything else.
Worked vignette: packaging a storage-engine benchmark
A submission proposes a write-optimized index validated on a telemetry trace and a synthetic
sweep.
- Ship the workload as a parameterized generator (append rate, append-to-scan ratio, key
distribution), not constants buried in a driver, so reviewers can vary the regime.
- Record the exact seed sequence and run count behind every throughput and latency-tail
figure; percentile claims are meaningless without them.
- Emit figures directly from logged runs so PDF and artifact numbers cannot drift.
- Include the ablation switch and the baseline configs so a reviewer can reproduce both
the effect and the fair comparison.
Single-blind and logistics anchors
- ICDE is single-blind: the artifact need not be anonymized — leave author names and history
in place; spend the effort on making it build and run.
- Assume, absent a formal badge track, that only the README and one entry script get opened;
design for that. If a badge/reproducibility process does run this edition, read its criteria
before packaging.
- Upload size limits and accepted formats vary by edition; verify against the current CMT
submission form.
Output format
[Artifact role] scored supplement / post-acceptance reproducibility / public archive
[Contents] <source / generators / baseline-configs / logs / claims-map>
[Turnkey check] <does run_small.sh reproduce the headline result? y/n>
[Isolability] <is the mechanism ablation runnable in the artifact? y/n>
[Hygiene] <secrets / caches / bloat removed>
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → ICDE-Skills/skills/icde-artifact-evaluation/SKILL.md
1---2name: icde-artifact-evaluation3description: Use when packaging IEEE ICDE code, data, workload generators, and logs as supplemental material whose availability reviewers score, and for any post-acceptance reproducibility or badge process the edition runs. Covers what a builder-heavy committee inspects first, making a data-systems benchmark turnkey, and single-blind packaging.4---567# ICDE Artifact Evaluation89Use this for evidence packaging around ICDE. ICDE expects authors to submit supplemental10material and **considers its availability in the evaluation**, so a strong artifact directly11raises the paper's floor even where no separate badge process runs. Confirm whether the current12edition also runs a post-acceptance reproducibility/badge track (待核实) before promising13evaluators anything.1415## Artifact plan1617- Decide what evidence a **builder** needs to believe the numbers: the system source, the18 workload generators, the baselines' configurations, the datasets or their construction19 scripts, seeds, and logs.20- Keep decision-critical evidence in the **paper or its figures**; the artifact demonstrates21 reproducibility, it does not replace the argument.22- Provide a **minimal reproduction map**: build steps, dependencies, hardware assumptions23 (especially the storage device), commands, expected outputs, runtime, and known24 nondeterminism sources.25- For restricted or proprietary data, give enough provenance and construction detail for a26 credible re-run without violating data-use terms.27- After acceptance, publish the **public, licensed, tagged** version whose commit produced the28 paper's numbers.2930## What ICDE evidence reviewers open first3132| Claim type | First artifact inspected | Common failure caught |33|---|---|---|34| Throughput/latency win | `run_small.sh` and the workload generator | Numbers cannot be regenerated; generator or seeds absent |35| "Mechanism causes the gain" | The ablation toggle in the code | The toggle does not exist; gain not isolable in the artifact |36| Baseline comparison | The baseline's config files | Baseline was untuned or run with defaults |37| Scale claim | The scale-factor sweep script | Only one scale factor is actually runnable |38| Cost/overhead claim | The instrumentation that measures cost | Cost is asserted but not measured anywhere |3940Because ICDE reviewers are builders, they will **re-run a small benchmark** far sooner than they41will provision a cluster — make `run_small.sh` reproduce the headline crossover on one machine42in minutes before polishing anything else.4344## Worked vignette: packaging a storage-engine benchmark4546A submission proposes a write-optimized index validated on a telemetry trace and a synthetic47sweep.4849- Ship the workload as a **parameterized generator** (append rate, append-to-scan ratio, key50 distribution), not constants buried in a driver, so reviewers can vary the regime.51- Record the **exact seed sequence and run count** behind every throughput and latency-tail52 figure; percentile claims are meaningless without them.53- Emit figures **directly from logged runs** so PDF and artifact numbers cannot drift.54- Include the **ablation switch** and the **baseline configs** so a reviewer can reproduce both55 the effect and the fair comparison.5657## Single-blind and logistics anchors5859- ICDE is single-blind: the artifact need **not** be anonymized — leave author names and history60 in place; spend the effort on making it build and run.61- Assume, absent a formal badge track, that only the **README and one entry script** get opened;62 design for that. If a badge/reproducibility process does run this edition, read its criteria63 before packaging.64- Upload size limits and accepted formats vary by edition; verify against the current CMT65 submission form.6667## Output format6869```text70[Artifact role] scored supplement / post-acceptance reproducibility / public archive71[Contents] <source / generators / baseline-configs / logs / claims-map>72[Turnkey check] <does run_small.sh reproduce the headline result? y/n>73[Isolability] <is the mechanism ablation runnable in the artifact? y/n>74[Hygiene] <secrets / caches / bloat removed>75[Fixes before upload] <ordered list>76```7778---7980**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `ICDE-Skills/skills/icde-artifact-evaluation/SKILL.md`
thedixitjain