brycewang-stanford HPCA Reproducibility
Use this to make an HPCA paper's results reconstructable — by a reviewer during the window, by an artifact evaluator after acceptance, and by you six months later when the environment has drifted. HPCA's artifact badges are IEEE reproducibility badges earned on a separate AE HotCRP, so build toward that pipeline, not the ACM one.
Pin the result chain
Every figure in the paper should trace back through an unbroken chain:
simulator commit (or machine + firmware)
-> configuration file (core, cache, DRAM, interconnect)
-> workload + input + sampled region
-> raw output logs
-> statistics script
-> the figure/table in the PDF
If any link is unpinned — a simulator built from "latest," a config edited by hand, a region chosen ad hoc — the result is not reproducible, only re-runnable by you.
One manifest per figure
Give each headline figure a manifest that names its commit, config, workload, region,
seed/trial count, and the exact command that regenerates it. A regen.sh per figure
that reproduces the artifact from raw logs turns "trust me" into "run this."
| Chain link | What to pin | Failure if unpinned |
|---|---|---|
| Simulator | Commit hash + local-patch diff | Silent behavior drift between builds |
| Config | Versioned file, not manual edits | Unreproducible operating point |
| Workload/region | Suite version + sampling recipe | Cherry-picked region suspicion |
| Statistics | Scripted, seeded | Numbers that cannot be regenerated |
| Silicon runs | Governor/turbo/SMT/NUMA + kernel/firmware | Noise mistaken for signal |
Record machine state for silicon
A real-hardware result is only reproducible with its host state captured: frequency governor, turbo, SMT, NUMA policy, kernel and firmware versions, and the trial count with dispersion. Archive it next to the logs, not in your memory.
Handle non-redistributable workloads
SPEC-class and other licensed suites cannot ship in the artifact. Provide the recipe and checksums to reconstruct them, plus at least one freely available workload that exercises the full pipeline so an evaluator without a license can still reproduce a headline result.
Keep the environment resurrectable
Between submission and the badge round, environments rot. Containerize or script the build, pin dependency versions, and do a cold-start resurrection drill on a clean machine before you promise a reviewer or evaluator anything. The single rebuttal/revision window is not the time to discover your simulator no longer builds.
Reproducibility checklist
[Result chain] every figure pinned commit->config->workload->region->stats? (Y/N)
[Manifests] one per headline figure with a regen script? (Y/N)
[Silicon state] governor/turbo/SMT/NUMA + kernel/firmware + trials archived? (Y/N/NA)
[Licensed workloads] recipe+checksums + one free workload provided? (Y/N/NA)
[Resurrection] cold-start build drill passed on a clean machine? (Y/N)
[Badge readiness] packaged for the IEEE AE HotCRP? (Y/N)
[Top gaps] <ordered>
Reopen the current artifact-evaluation page for the badge set and calendar before relying on any AE detail here — the AE mechanics are per-edition.
Source: brycewang-stanford/Awesome-Journal-Skills → HPCA-Skills/skills/hpca-reproducibility/SKILL.md