thedixitjain PPoPP Reproducibility
Build the reproducibility story that a parallel-performance paper needs. Unlike a deterministic algorithm, a PPoPP result depends on the machine, the topology, and the run conditions — and an evaluator or a future reader will not have your exact node. Reproducibility here means someone else can rebuild the environment and recover the scaling trend and the relative comparison, even when their absolute numbers differ. Pin everything you can at run time; you cannot reconstruct it after the machine is released.
The environment description reviewers actually re-run
State enough that a stranger could stand up the same experiment:
[CPU] exact model, sockets, cores/threads, base/turbo policy, NUMA node layout
[Memory] size, channels, speed; per-socket bandwidth if it bounds you
[GPU] model, count, driver + CUDA/ROCm version, connection (PCIe/NVLink)
[Interconnect] for multi-node: fabric and topology
[Software] OS + kernel, compiler + version + exact flags, libraries + versions, allocator
[Runtime] thread count(s), pinning/affinity policy, scheduler settings, env vars
An evaluation whose machine is described only as "a Linux server" is the one reviewers trust least and cannot reproduce.
Control the sources of parallel non-determinism
- Thread pinning / affinity. Pin threads to cores and state the policy; unpinned runs migrate
and produce unrepeatable numbers. Document NUMA placement (e.g.,
numactl, first-touch policy). - Warm-up. Discard JIT, cache, and allocator warm-up; report the steady-state measurement protocol and how many iterations you drop.
- Seeds. Fix and record seeds for any randomized workload, input generator, or scheduler.
- Frequency scaling. Disable turbo/DVFS or report that it is on; otherwise speedups drift with temperature and neighbors.
- Isolation. Run on a quiescent machine with no co-tenants; note it.
- Repeats. Multiple runs with reported variance — reproducibility includes the spread, not just the mean.
Reproduce the trend, not just the number
Because evaluators have different hardware, design the package so the conclusion survives a change of machine:
- Provide a scaled-down configuration whose curve shape (linear region, saturation point) matches the paper, runnable on modest hardware.
- State the minimum configuration on which the claim still holds, and the tolerance within which you consider a result reproduced (e.g., "monotone speedup to the machine's core count," or "within 10% at matched core counts").
- Separate machine-dependent absolute throughput from machine-independent claims (relative speedup, scaling shape, correctness).
Data and workloads
- Ship or point to the exact inputs (graphs, matrices, traces) with sizes and provenance, not just a generator script — or ship the generator with its seed.
- For real-application workloads, document versions and how they were built; a benchmark suite name without a version pin is not reproducible.
Correctness reproducibility
- If a model checker or race detector supports your correctness claim, ship its configuration and the harness so the check itself can be re-run, not just described.
Pre-submission reproducibility pass
[Fresh checkout] build from a clean clone in the pinned container -> succeeds?
[Blind re-run] a colleague re-runs a scaling figure on a *different* machine -> trend matches?
[Provenance] every number in the paper traces to a script + a recorded machine config?
[Anonymity] the review-time package hides machine names, accounts, and personal repos?
Output format
[Environment] CPU/GPU/memory/interconnect/software/runtime fully stated? gaps: <...>
[Non-determinism] pinning, warm-up, seeds, frequency, isolation, repeats controlled? list gaps
[Trend portability] scaled config + minimum config + tolerance stated? yes/no
[Data] exact inputs or seeded generator shipped/pinned? yes/no
[Pass result] fresh-checkout build + cross-machine trend match? yes/no
Source: brycewang-stanford/Awesome-Journal-Skills → PPoPP-Skills/skills/ppopp-reproducibility/SKILL.md