brycewang-stanford MICRO Reproducibility
In microarchitecture research the "experiment" is usually a software artifact pretending to be hardware. That makes results perfectly reproducible in principle — and silently unreproducible in practice, because a simulator commit, a config diff, or a trace-generation flag changed between the headline run and the camera-ready. This skill installs the record-keeping that prevents that.
What must be pinned, by evidence type
| Evidence | Pin these or the number is unrecoverable |
|---|---|
| Cycle-level simulation | Simulator name + exact commit hash; full config files (not "8-wide OoO" prose); patch set applied; host compiler for the simulator build |
| Workload traces | Source binary + compiler flags; trace tool + version; SimPoint interval size, K, and seed; warmup instructions per region |
| Power / area estimates | McPAT/CACTI/Accelergy version; technology node; the XML/cfg input actually fed in; any manual scaling factors |
| RTL results | HDL commit; synthesis tool + version; target library/PDK; constraints file; which corners were run |
| FPGA / silicon | Board/chip stepping; bitstream hash or firmware; frequency; measurement method for power (rail vs counter model) |
| DRAM behavior | Ramulator/DRAMsim3 version + timing config (device datasheet parameters) |
The rule of thumb: a result is reproducible when a stranger can regenerate the exact bar in Fig. 3 from the manifest alone, without asking you anything.
The run manifest
Emit one manifest per experiment batch, generated by the launch script — never written by hand after the fact:
# results/2026-04-02_headline/manifest.yaml
simulator: {name: champsim, commit: 3fa9c21, patches: [prefetch-fix.diff]}
config: {file: configs/8w_ooo_baseline.json, sha256: 9be1...}
mechanism: {file: configs/ours_32kb.json, knobs: {table_entries: 2048, tag_bits: 12}}
workloads: {suite: spec2017_rate1, traces: traces/v3/, simpoints: k30_int200M_seed7,
warmup_M: 200, detailed_M: 1000}
power: {tool: mcpat, version: "1.3", node_nm: 22, input: mcpat/ours.xml}
host: {machine: lab-node-14, compiler: gcc-13.2}
outputs: {raw: stats/, figures: fig3_headline.pdf, script: plot_headline.py}
Commit the manifest with the raw stats files. Figures are regenerated from stats by a script named in the manifest — a figure that cannot be regenerated does not go in the paper.
Disclosure lines the paper itself owes
The methodology section (see micro-writing-style, ~1 page) must state, in text:
- Simulator and version, with your modifications summarized honestly ("we added a cycle-accurate model of X; the interconnect remains the default model").
- The full baseline configuration in a table (pipeline width, ROB, caches with latencies, predictor, prefetchers, DRAM timing).
- Sampling recipe: how regions were chosen, warmup, and measured instruction counts.
- Power/area model, node, and the sentence "these are model estimates" — the community's tools (CACTI, McPAT — both MICRO papers themselves) are respected but known to be approximations, and pretending otherwise reads as naivety.
Determinism and drift checks
- Two identical launches must produce bit-identical stats; if the simulator has any nondeterminism (thread interleaving in parallel sims), pin it or report run count and spread.
- Re-run the baseline after every simulator patch — a patch that changes baseline IPC by 0.5% invalidates every relative number already collected.
- Keep a
results-ledger.md: date, manifest path, headline geomean, what changed. Rebuttal season (June) will ask "what config produced Fig. 7?" — from three months earlier; the ledger answers in minutes, memory does not.
Validating the instrument itself
Reproducibility of a wrong number is worthless; record the evidence that the simulator models what the paper says it models:
- Cross-check the baseline against published behavior (see
micro-experiments): store the comparison table — suite, metric, published range, your value — in the repo, not just in someone's memory. - Unit-test the mechanism model. Hand-construct access sequences with known correct outcomes (a burst that must train the table, a pattern that must not) and assert the simulator's counters match. These micro-tests are the first thing an AE evaluator can run and the best evidence the model is not a stats bug.
- Record known fidelity gaps in a
LIMITS.md: default interconnect model, simplified TLB, unmodeled prefetcher interactions. The paper's claims must stay inside these limits; writing them down keeps deadline-week edits honest.
Anonymity interaction (submission phase)
MICRO 2026 requires artifact links to be fully anonymized or removed. Before the
April upload, sweep the repo you intend to link: git author fields, hostnames in
manifests (lab-node-14 is fine; patt-lab-utexas-14 is not), cluster paths,
funding strings in READMEs. The manifest discipline above makes this sweep
mechanical rather than archaeological.
Storage economics of keeping everything
Raw cycle-level stats for a full paper (dozens of configs × dozens of workloads × sweeps) usually fit in single-digit gigabytes — trivially archivable, and the cheapest insurance in research. Policy: keep all raw stats files, forever, compressed; regenerate figures on demand. Delete only simulator checkpoints and intermediate trace files that the recipes can rebuild. When a reviewer asks in June about a bar from a January run, or AE asks in August, or a follow-up paper asks in two years, the manifest + stats pair answers; the alternative is re-simulating weeks of compute under deadline pressure.
Output format
[Pinning audit] per evidence type: pinned / partially pinned / loose (list)
[Manifest coverage] experiments with manifests: N/M
[Figure regen] all paper figures scripted from raw stats: yes / no (list unscripted)
[Methodology text] simulator+mods / baseline table / sampling / model-estimate
disclosure: present-absent each
[Determinism] identical-rerun check passed: yes / no / untested
[Ledger] results-ledger current through: <date>
[Anonymity sweep] repo clean for anonymized linking: yes / issues listed