MICRO Experiments
MICRO's evaluation culture is instrument-centric: the community knows exactly what
each measurement tool can and cannot prove, and reviewers score the match between
claim and instrument before they look at the numbers.
The instrument ladder
| Instrument |
Proves |
Cannot prove |
Typical tools |
| Analytical / trace model |
First-order potential, limit studies |
Interaction effects, timing |
custom, trace-driven models |
| Cycle-level simulation |
Relative performance of mechanisms |
Absolute wall-clock, physical cost |
gem5, ChampSim, Sniper, Ramulator/DRAMsim3 |
| Power/area models on top of sim |
Energy and area trends |
Sign-off-quality numbers |
McPAT, CACTI, Accelergy |
| RTL + synthesis |
Timing closure, real area/power at a node |
Full-system performance |
Verilog/Chisel + synthesis flow |
| FPGA prototype |
Functional correctness at scale, OS interaction |
ASIC frequency/power |
FireSim, custom boards |
| Silicon measurement |
Everything, for that one chip |
Generality across designs |
perf counters, power rails |
Rule: claim one rung below your strongest instrument. Cycle-level simulation
plus McPAT supports "reduces memory-stall cycles by X% with ~Y mm² estimated
overhead"; it does not support "improves datacenter TCO." The lineage here is the
venue's own: CACTI 6.0 (MICRO 2007) and McPAT (MICRO 2009) were published at MICRO
precisely because the community polices modeling fidelity.
Baseline construction — where most rejections start
- The baseline core must be configured like a current product, not a textbook
default: wide OoO, capable branch predictor, competent multi-stream prefetcher,
realistic DRAM timing. A win over gem5's out-of-the-box config is a non-result.
- Include the best prior mechanism in your exact category, re-implemented in
your simulator with its published parameters — not just numbers copied from its
paper under a different config.
- Add an idealized upper bound (oracle predictor, infinite table) so readers see
what fraction of the headroom you capture.
- If your mechanism uses N KB of extra storage, give the baseline the same N KB as
additional cache/predictor capacity in at least one comparison — the
"iso-storage" check reviewers ask for in rebuttal anyway.
Workloads
- Name the suite and the subsetting rule: SPEC CPU2017 (all, or a stated-criterion
subset), PARSEC/GAP for multithreaded, MLPerf or named DNNs for accelerators,
plus the domain traces your mechanism targets.
- Disclose sampling methodology: SimPoint regions, warmup lengths, instruction
budgets per region. Cherry-picked "representative regions" without a stated
selection rule are a rebuttal magnet.
- Report per-workload bars plus geomean — never arithmetic means of speedups,
and never geomean-only (it hides the regressions reviewers will hunt for).
# the only defensible summary statistic for speedups
from math import prod
def geomean(xs):
return prod(xs) ** (1.0 / len(xs))
speedups = per_workload_ipc_new / per_workload_ipc_base # elementwise
print(f"geomean {geomean(list(speedups)):.3f}, "
f"min {min(speedups):.3f} ({worst_workload}), "
f"regressions: {(speedups < 1.0).sum()} of {len(speedups)}")
The ablation and sensitivity contract
Every design decision named in the mechanism section owes the evaluation section a
figure or table row:
- Ablate each component (drop the filter, halve the table, disable the guard)
to show each earns its area.
- Sweep the structural parameters: table sizes, associativities, thresholds,
core counts, LLC capacities, DRAM bandwidth. The mechanism should degrade
gracefully at the sweep edges — cliffs demand explanation in the text.
- Stress adversarially: the workload class the mechanism should not help, the
access pattern designed to defeat it. Reporting a bounded loss builds more trust
than an unbroken win column.
Simulation-length and validation sanity
Two credibility checks reviewers apply that authors often skip:
- Enough simulated instructions per region. Sub-100M-instruction detailed
windows on memory-bound workloads mostly measure the warmup transient. State
warmup and detailed lengths, and show at least once that doubling them does not
move the headline.
- Baseline validation against published numbers. Before trusting relative
results, show your baseline's absolute behavior is sane: IPC or MPKI within
the published envelope for the same suite and a comparable config. A baseline
whose branch predictor achieves 2x the published MPKI of the modeled design
invalidates the comparison silently.
For multicore results, disclose how heterogeneity was handled: mix construction
rule, per-mix repetitions, and whether throughput is weighted speedup, harmonic
mean, or raw IPC sum — each answers a different question and reviewers check that
the metric matches the claim (fairness claims need a fairness metric).
Overhead accounting checklist
Output format
[Instrument] <rung used> — claim height matches: yes / no (quote the overclaim)
[Baseline strength] product-like config / best-prior reimplemented / oracle bound /
iso-storage check: present-absent each
[Workloads] suite + subset rule + sampling disclosure: complete / gaps listed
[Summary stats] per-workload + geomean + regression count: yes / no
[Ablations] each mechanism component covered: list of unablated components
[Sensitivity] parameters swept vs parameters hardcoded
[Overheads] storage / area / power / latency / energy: quantified-missing each
1---2name: micro-experiments3description: Use when designing or auditing the evaluation of a MICRO paper — choosing the right instrument on the ladder from analytical model to cycle-level simulator to RTL to silicon, tuning baselines the PC will respect, selecting workload suites, running ablations and sensitivity sweeps, and reporting geomeans with full overhead accounting.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/micro-experiments6---78# MICRO Experiments910MICRO's evaluation culture is instrument-centric: the community knows exactly what11each measurement tool can and cannot prove, and reviewers score the *match between12claim and instrument* before they look at the numbers.1314## The instrument ladder1516| Instrument | Proves | Cannot prove | Typical tools |17|---|---|---|---|18| Analytical / trace model | First-order potential, limit studies | Interaction effects, timing | custom, trace-driven models |19| Cycle-level simulation | Relative performance of mechanisms | Absolute wall-clock, physical cost | gem5, ChampSim, Sniper, Ramulator/DRAMsim3 |20| Power/area models on top of sim | Energy and area trends | Sign-off-quality numbers | McPAT, CACTI, Accelergy |21| RTL + synthesis | Timing closure, real area/power at a node | Full-system performance | Verilog/Chisel + synthesis flow |22| FPGA prototype | Functional correctness at scale, OS interaction | ASIC frequency/power | FireSim, custom boards |23| Silicon measurement | Everything, for that one chip | Generality across designs | perf counters, power rails |2425Rule: **claim one rung below your strongest instrument.** Cycle-level simulation26plus McPAT supports "reduces memory-stall cycles by X% with ~Y mm² estimated27overhead"; it does not support "improves datacenter TCO." The lineage here is the28venue's own: CACTI 6.0 (MICRO 2007) and McPAT (MICRO 2009) were published *at MICRO*29precisely because the community polices modeling fidelity.3031## Baseline construction — where most rejections start3233- The baseline core must be **configured like a current product**, not a textbook34 default: wide OoO, capable branch predictor, competent multi-stream prefetcher,35 realistic DRAM timing. A win over gem5's out-of-the-box config is a non-result.36- Include the **best prior mechanism in your exact category**, re-implemented in37 your simulator with its published parameters — not just numbers copied from its38 paper under a different config.39- Add an **idealized upper bound** (oracle predictor, infinite table) so readers see40 what fraction of the headroom you capture.41- If your mechanism uses N KB of extra storage, give the baseline the same N KB as42 additional cache/predictor capacity in at least one comparison — the43 "iso-storage" check reviewers ask for in rebuttal anyway.4445## Workloads4647- Name the suite and the subsetting rule: SPEC CPU2017 (all, or a stated-criterion48 subset), PARSEC/GAP for multithreaded, MLPerf or named DNNs for accelerators,49 plus the domain traces your mechanism targets.50- Disclose sampling methodology: SimPoint regions, warmup lengths, instruction51 budgets per region. Cherry-picked "representative regions" without a stated52 selection rule are a rebuttal magnet.53- Report **per-workload bars plus geomean** — never arithmetic means of speedups,54 and never geomean-only (it hides the regressions reviewers will hunt for).5556```python57# the only defensible summary statistic for speedups58from math import prod59def geomean(xs):60 return prod(xs) ** (1.0 / len(xs))61speedups = per_workload_ipc_new / per_workload_ipc_base # elementwise62print(f"geomean {geomean(list(speedups)):.3f}, "63 f"min {min(speedups):.3f} ({worst_workload}), "64 f"regressions: {(speedups < 1.0).sum()} of {len(speedups)}")65```6667## The ablation and sensitivity contract6869Every design decision named in the mechanism section owes the evaluation section a70figure or table row:7172- **Ablate** each component (drop the filter, halve the table, disable the guard)73 to show each earns its area.74- **Sweep** the structural parameters: table sizes, associativities, thresholds,75 core counts, LLC capacities, DRAM bandwidth. The mechanism should degrade76 gracefully at the sweep edges — cliffs demand explanation in the text.77- **Stress adversarially:** the workload class the mechanism should *not* help, the78 access pattern designed to defeat it. Reporting a bounded loss builds more trust79 than an unbroken win column.8081## Simulation-length and validation sanity8283Two credibility checks reviewers apply that authors often skip:8485- **Enough simulated instructions per region.** Sub-100M-instruction detailed86 windows on memory-bound workloads mostly measure the warmup transient. State87 warmup and detailed lengths, and show at least once that doubling them does not88 move the headline.89- **Baseline validation against published numbers.** Before trusting relative90 results, show your baseline's absolute behavior is sane: IPC or MPKI within91 the published envelope for the same suite and a comparable config. A baseline92 whose branch predictor achieves 2x the published MPKI of the modeled design93 invalidates the comparison silently.9495For multicore results, disclose how heterogeneity was handled: mix construction96rule, per-mix repetitions, and whether throughput is weighted speedup, harmonic97mean, or raw IPC sum — each answers a different question and reviewers check that98the metric matches the claim (fairness claims need a fairness metric).99100## Overhead accounting checklist101102- [ ] Storage: bits/entry × entries, totaled in KB, per core and shared.103- [ ] Area and power: model named with version (e.g., McPAT vX at Ynm), numbers104 labeled as estimates.105- [ ] Latency: added pipeline stages or access-path cycles; off critical path106 claims justified.107- [ ] Energy: dynamic + leakage deltas, not just "negligible."108- [ ] Complexity: verification surface, new SRAM ports, wiring — one honest109 paragraph.110111## Output format112113```text114[Instrument] <rung used> — claim height matches: yes / no (quote the overclaim)115[Baseline strength] product-like config / best-prior reimplemented / oracle bound /116 iso-storage check: present-absent each117[Workloads] suite + subset rule + sampling disclosure: complete / gaps listed118[Summary stats] per-workload + geomean + regression count: yes / no119[Ablations] each mechanism component covered: list of unablated components120[Sensitivity] parameters swept vs parameters hardcoded121[Overheads] storage / area / power / latency / energy: quantified-missing each122```