Mlsys Reproducibility

MLSys Reproducibility

Use this while experiments are still running — reproducibility at this venue is a measurement-design property, not a packaging afterthought. An MLSys claim is typically "system A beats system B by X% on workload W on hardware H," and every one of those four variables can silently drift. The venue's culture (badge-based artifact evaluation, the MLPerf benchmark lineage published in its own proceedings) means reviewers assume performance numbers will eventually be re-measured by someone else.

Two kinds of nondeterminism — control them separately

Papers routinely fix seeds meticulously while leaving thermal state and placement uncontrolled — backwards for a performance paper, where systems noise usually dwarfs seed effects on latency numbers.

The environment pin — deeper than requirements.txt

A latency claim depends on layers a Python lockfile never sees. Record all of them:

• Hardware: GPU/accelerator model and count, CPU, memory, interconnect (NVLink/PCIe generation, NIC), storage class.
• System: driver version, CUDA/ROCm version, container image digest, kernel version.
• Framework: exact framework build, compilation flags, graph/eager mode, precision (FP16/BF16/FP8/INT4), and any autotuning caches — a warm autotuner cache can fake a speedup that a fresh machine cannot reproduce.
• Serving stack: batch policy, concurrency limits, admission control settings.

Measurement harness discipline

import time, statistics

def measure(step, warmup=20, trials=200):
    for _ in range(warmup):          # exclude JIT, autotuning, cache-fill effects
        step()
    xs = []
    for _ in range(trials):
        t0 = time.perf_counter()
        step()                       # synchronize accelerator inside step()
        xs.append(time.perf_counter() - t0)
    xs.sort()
    return {
        "p50": xs[len(xs)//2],
        "p99": xs[int(len(xs)*0.99)],
        "mean": statistics.fmean(xs),
        "stdev": statistics.stdev(xs),
        "trials": trials,
    }

• Never report a single run for any latency or throughput number; state trial counts and either stdev or percentile spread in every table caption.
• Report tails (p95/p99) for anything serving-shaped; means alone hide the behavior systems reviewers care about most.
• Synchronize accelerators before timestamps — async launch makes GPUs look infinitely fast in naive harnesses.
• Interleave A/B trials (ABABAB, not AAABBB) so thermal drift and co-tenant noise hit both systems equally.
• Keep raw measurement logs; tables should be generated from logs by script, so the paper, the artifact, and reality cannot diverge.

Disclosure floor for the paper itself

• Workload identity: exact models, datasets or traces, sequence-length/request-rate distributions, and how the workload was chosen (a named benchmark family beats a bespoke workload for credibility — reviewers know the MLPerf-style conventions).
• Baseline versions and their tuning budget, stated symmetrically with your system's.
• Total compute consumed and, where the paper argues cost efficiency, the price basis ($/GPU-hour source and date) behind any dollar figures.
• Energy numbers, if claimed, with the measurement method (whole-node meter vs. software counters) — the two can disagree wildly.
• What was not controlled, honestly: shared cluster, single hardware family, one precision mode. A scoped claim survives re-measurement; an unscoped one does not.

Common measurement bugs this venue catches

Each of these has sunk real performance claims; check for them before a reviewer does.

• Warm-cache flattery: benchmarking after the dataset, weights, or autotuner cache is hot, while the baseline runs cold. Symmetrize or report both regimes.
• Async mirage: timing GPU work without device synchronization, measuring launch latency instead of execution.
• Batch-mismatch comparisons: your system at its best batch size versus the baseline at its default — a tuning-parity violation wearing a measurement disguise.
• Coordinated omission: measuring latency only for requests the system accepted, while it sheds load; report drop/timeout rates next to every latency figure.
• Averaging across heterogeneous workloads: a single mean over workloads of wildly different scales lets one workload buy the headline; report per-workload numbers.
• Power-state contamination: comparing runs taken at different GPU clock or thermal states; record clocks and lock them where the platform allows.

Pre-submission reproducibility drill

1. Fresh-clone the repo on a machine that has never run the project; follow only the README. Every undocumented step found here is a future AE failure.
2. Regenerate the two most important tables from raw logs with one command each.
3. Diff regenerated numbers against the PDF; investigate any discrepancy beyond stated variance — this drill catches stale-table bugs that reviewers cannot, but artifact evaluators will.
4. Record the wall-clock and dollar cost of the full reproduction; put it in the appendix so others can budget.

Cycle-volatility warning

Whether MLSys requires a reproducibility checklist or statement at submission time is a per-cycle decision that could not be verified for 2026 (待核实) — check the current CFP and OpenReview form fields rather than assuming either way. Artifact-evaluation badge mechanics live in mlsys-artifact-evaluation.

Output format

[Claim under audit] <system-vs-baseline, workload, hardware>
[Environment pin] <hardware/driver/container/framework/precision status>
[Noise controls] <warmup/trials/interleaving/placement/tails reported?>
[Disclosure gaps] <workload provenance/baseline tuning/compute/cost/energy>
[Drill result] <fresh-machine reproduction outcome + cost>
[Fixes] <ordered, cheapest-first>

Source: brycewang-stanford/Awesome-Journal-Skills → MLSys-Skills/skills/mlsys-reproducibility/SKILL.md