Asplos Reproducibility

ASPLOS Reproducibility

Systems results decay fast: a kernel update, a microcode revision, or a silently changed simulator default can move numbers by more than the paper's claimed margin. Reproducibility work at ASPLOS is therefore state capture — recording the full machine, model, and toolchain state behind every figure — done while the experiments run, not reconstructed at camera-ready time. It also front-loads artifact evaluation: the badge criteria (asplos-artifact-evaluation) are exactly a demand that this state capture exists and works.

The state ledger

Maintain one ledger row per experimental platform, committed alongside results:

Scripted capture beats remembered capture

Run at the start of every measurement session; store output next to the data:

#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null   # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null              # FPGA bitstream identity

The hardware-access problem, named honestly

ASPLOS artifacts often need hardware an independent evaluator will not have. The honest pattern is a three-tier availability statement drafted at submission time:

1. Repeatable anywhere: simulator experiments and analysis scripts — full configs and one command per figure.
2. Repeatable with named hardware: the exact platform requirements (board, expander, CPU family), plus what to expect if the evaluator's part differs.
3. Not independently repeatable: results on lab-only or pre-production hardware — say so, and provide either supervised access, raw logs with the analysis pipeline, or a scaled-down proxy. Silence here reads as concealment; a stated limitation reads as engineering.

Claim-preservation, not number-worship

State which conclusions should survive environmental drift and which are environment-specific: "the ordering of policies is stable across kernels 6.6-6.9; absolute runtimes are not." This single sentence pattern prevents the most common failed-reproduction dispute — an evaluator matching your ordering but not your absolute numbers and calling it a failure.

Timing across the ASPLOS cycle

• Before September 9: ledger current; capture script in the repo; availability tiers drafted (they inform the paper's own text).
• Response window: the ledger is your defense when a reviewer doubts a number — you can state the exact conditions instead of hand-waving.
• Major Revision: re-run under the captured original state where possible; where the environment has drifted, disclose the drift in the change note.
• After acceptance: the ledger becomes the Artifact Appendix's dependency section nearly verbatim; AE calendars for 2027 were 待核实 at pack-check time, so confirm dates when notified.

One command per figure

The internal gold standard that makes everything downstream cheap: every figure and table in the paper regenerates from a single committed command that reads raw results and emits the exact plot. It catches stale-figure bugs before submission, turns response-window questions into lookups, and becomes the Reproducible-badge run script with a rename. Institute it at the first result, when it costs minutes — retrofitting it at camera-ready costs days.

Trace and dataset provenance

Workload inputs decay independently of code. For each trace or dataset, record origin (public suite version, generated-by script + seed, or production source), preprocessing steps as scripts rather than prose, and a checksum of the exact bytes used. Production traces that cannot be released need a characterization (rate, skew, working-set curves) plus a matched synthetic generator committed to the repo — this is also the anonymity-safe form for submission, since a raw trace can identify its owner.

When numbers drift between submission and revision

The Major Revision window arrives months after the original runs, and environments drift. Protocol:

1. Re-run a sentinel subset (three representative experiments) under the captured original state before starting revision work; if the sentinels reproduce, extend confidently.
2. If they do not, bisect the ledger — kernel, microcode, simulator commit — until the moved variable is found; the ledger exists for exactly this moment.
3. Disclose in the change note which results were re-collected and under what changed conditions, and re-state the claim-preservation sentence for the new environment. Silent regeneration of all numbers invites a reviewer to ask which version was real.

Output format

[Ledger coverage] platforms with complete rows: N/N · gaps listed
[Capture automation] script committed + outputs stored with data: Y/N
[Simulator pinning] commit + configs + region method + warm-up recorded: Y/N
[Availability tiers] anywhere / named-hardware / not-repeatable — each populated
[Claim preservation] drift-stable vs environment-specific conclusions stated: Y/N
[Badge readiness] which badges the current package could already earn

Source: brycewang-stanford/Awesome-Journal-Skills → ASPLOS-Skills/skills/asplos-reproducibility/SKILL.md