Micro Reproducibility

MICRO Reproducibility

In microarchitecture research the "experiment" is usually a software artifact pretending to be hardware. That makes results perfectly reproducible in principle — and silently unreproducible in practice, because a simulator commit, a config diff, or a trace-generation flag changed between the headline run and the camera-ready. This skill installs the record-keeping that prevents that.

What must be pinned, by evidence type

The rule of thumb: a result is reproducible when a stranger can regenerate the exact bar in Fig. 3 from the manifest alone, without asking you anything.

The run manifest

Emit one manifest per experiment batch, generated by the launch script — never written by hand after the fact:

# results/2026-04-02_headline/manifest.yaml
simulator: {name: champsim, commit: 3fa9c21, patches: [prefetch-fix.diff]}
config:    {file: configs/8w_ooo_baseline.json, sha256: 9be1...}
mechanism: {file: configs/ours_32kb.json, knobs: {table_entries: 2048, tag_bits: 12}}
workloads: {suite: spec2017_rate1, traces: traces/v3/, simpoints: k30_int200M_seed7,
            warmup_M: 200, detailed_M: 1000}
power:     {tool: mcpat, version: "1.3", node_nm: 22, input: mcpat/ours.xml}
host:      {machine: lab-node-14, compiler: gcc-13.2}
outputs:   {raw: stats/, figures: fig3_headline.pdf, script: plot_headline.py}

Commit the manifest with the raw stats files. Figures are regenerated from stats by a script named in the manifest — a figure that cannot be regenerated does not go in the paper.

Disclosure lines the paper itself owes

The methodology section (see micro-writing-style, ~1 page) must state, in text:

• Simulator and version, with your modifications summarized honestly ("we added a cycle-accurate model of X; the interconnect remains the default model").
• The full baseline configuration in a table (pipeline width, ROB, caches with latencies, predictor, prefetchers, DRAM timing).
• Sampling recipe: how regions were chosen, warmup, and measured instruction counts.
• Power/area model, node, and the sentence "these are model estimates" — the community's tools (CACTI, McPAT — both MICRO papers themselves) are respected but known to be approximations, and pretending otherwise reads as naivety.

Determinism and drift checks

• Two identical launches must produce bit-identical stats; if the simulator has any nondeterminism (thread interleaving in parallel sims), pin it or report run count and spread.
• Re-run the baseline after every simulator patch — a patch that changes baseline IPC by 0.5% invalidates every relative number already collected.
• Keep a results-ledger.md: date, manifest path, headline geomean, what changed. Rebuttal season (June) will ask "what config produced Fig. 7?" — from three months earlier; the ledger answers in minutes, memory does not.

Validating the instrument itself

Reproducibility of a wrong number is worthless; record the evidence that the simulator models what the paper says it models:

• Cross-check the baseline against published behavior (see micro-experiments): store the comparison table — suite, metric, published range, your value — in the repo, not just in someone's memory.
• Unit-test the mechanism model. Hand-construct access sequences with known correct outcomes (a burst that must train the table, a pattern that must not) and assert the simulator's counters match. These micro-tests are the first thing an AE evaluator can run and the best evidence the model is not a stats bug.
• Record known fidelity gaps in a LIMITS.md: default interconnect model, simplified TLB, unmodeled prefetcher interactions. The paper's claims must stay inside these limits; writing them down keeps deadline-week edits honest.

Anonymity interaction (submission phase)

MICRO 2026 requires artifact links to be fully anonymized or removed. Before the April upload, sweep the repo you intend to link: git author fields, hostnames in manifests (lab-node-14 is fine; patt-lab-utexas-14 is not), cluster paths, funding strings in READMEs. The manifest discipline above makes this sweep mechanical rather than archaeological.

Storage economics of keeping everything

Raw cycle-level stats for a full paper (dozens of configs × dozens of workloads × sweeps) usually fit in single-digit gigabytes — trivially archivable, and the cheapest insurance in research. Policy: keep all raw stats files, forever, compressed; regenerate figures on demand. Delete only simulator checkpoints and intermediate trace files that the recipes can rebuild. When a reviewer asks in June about a bar from a January run, or AE asks in August, or a follow-up paper asks in two years, the manifest + stats pair answers; the alternative is re-simulating weeks of compute under deadline pressure.

Output format

[Pinning audit] per evidence type: pinned / partially pinned / loose (list)
[Manifest coverage] experiments with manifests: N/M
[Figure regen] all paper figures scripted from raw stats: yes / no (list unscripted)
[Methodology text] simulator+mods / baseline table / sampling / model-estimate
                   disclosure: present-absent each
[Determinism] identical-rerun check passed: yes / no / untested
[Ledger] results-ledger current through: <date>
[Anonymity sweep] repo clean for anonymized linking: yes / issues listed

Source: brycewang-stanford/Awesome-Journal-Skills → MICRO-Skills/skills/micro-reproducibility/SKILL.md