brycewang-stanford HPCA Artifact Evaluation
Use this to prepare an HPCA artifact for the badge round. HPCA's artifact evaluation
is voluntary, post-acceptance, and run on a separate HotCRP
(hpca<year>ae.hotcrp.com) under IEEE reproducibility badging — not the ACM
Artifact Review and Badging policy the ACM-touched siblings use. Package for the IEEE
pipeline and for an evaluator who has never seen your infrastructure.
Scope the reproducibility tiers
Not every claim can be reproduced cheaply. Decide, per headline result, what an evaluator can realistically achieve and say so:
- Full reproduction — the evaluator regenerates the headline numbers from source. The strongest badge target; reserve it for results whose pipeline you can shrink to fit an evaluator's budget.
- Regeneration from logs — the evaluator rebuilds figures from provided raw output when a full run is too long or needs licensed workloads.
- Inspection — the evaluator confirms the code implements the mechanism when hardware or license constraints block execution.
State each result's tier in the artifact README so evaluators are not surprised.
Package for a cold-start evaluator
The evaluator has no license to your workloads, no copy of your machine, and a fixed time budget. Design for that:
| Obstacle | What to ship |
|---|---|
| No workload license | Recipe + checksums, plus one free workload that runs the full pipeline |
| No access to your machine | Container/script that builds on a clean host; pinned dependencies |
| Long simulations | Reduced-input variants of the headline runs, with measured runtimes |
| Silicon-only results | Captured machine-state logs + an inspection path when re-running is impossible |
| Unclear entry point | A top-level run.sh that reproduces one headline figure end to end |
Budget the evaluator's wall-clock
An artifact that "reproduces everything" in three days of compute will not be fully reproduced. Provide reduced-input variants of the headline experiments with stated per-step runtimes, so an evaluator can plan and finish inside the AE window. Put the expensive full runs behind a clearly labeled optional path.
Mirror ↔ release
The anonymized artifact mirror used during review becomes the de-anonymized public release at the badge round. De-anonymize deliberately: restore author names, add the real repository, and check that nothing was left blinded that should now be open — and nothing personal was left in that should not.
Packaging pass
1. Per-result tier declared (full / regeneration / inspection)
2. Cold-start build works on a clean host (container or script, pinned deps)
3. One free workload exercises the full pipeline end to end
4. Reduced-input variants with measured per-step runtimes provided
5. Licensed workloads reduced to recipe + checksums
6. run.sh reproduces one headline figure without hand-holding
7. De-anonymized for release; README states tiers and expected runtimes
Output format
[AE readiness] Ready / Needs work / Not ready
[Tiers] results with a declared tier / total
[Cold start] clean-host build passes? (Y/N)
[Budget] reduced variants + runtimes stated? (Y/N)
[Badge target] full-reproduction results / total
[Top gaps] <ordered>
Reopen the current AE page for the badge set, the calendar, and the submission mechanics — the IEEE badge names and AE timeline are per-edition.
Source: brycewang-stanford/Awesome-Journal-Skills → HPCA-Skills/skills/hpca-artifact-evaluation/SKILL.md