PPoPP Reproducibility
Build the reproducibility story that a parallel-performance paper needs. Unlike a deterministic
algorithm, a PPoPP result depends on the machine, the topology, and the run conditions — and an
evaluator or a future reader will not have your exact node. Reproducibility here means someone else
can rebuild the environment and recover the scaling trend and the relative comparison, even when
their absolute numbers differ. Pin everything you can at run time; you cannot reconstruct it after
the machine is released.
The environment description reviewers actually re-run
State enough that a stranger could stand up the same experiment:
[CPU] exact model, sockets, cores/threads, base/turbo policy, NUMA node layout
[Memory] size, channels, speed; per-socket bandwidth if it bounds you
[GPU] model, count, driver + CUDA/ROCm version, connection (PCIe/NVLink)
[Interconnect] for multi-node: fabric and topology
[Software] OS + kernel, compiler + version + exact flags, libraries + versions, allocator
[Runtime] thread count(s), pinning/affinity policy, scheduler settings, env vars
An evaluation whose machine is described only as "a Linux server" is the one reviewers trust least
and cannot reproduce.
Control the sources of parallel non-determinism
- Thread pinning / affinity. Pin threads to cores and state the policy; unpinned runs migrate
and produce unrepeatable numbers. Document NUMA placement (e.g.,
numactl, first-touch policy).
- Warm-up. Discard JIT, cache, and allocator warm-up; report the steady-state measurement
protocol and how many iterations you drop.
- Seeds. Fix and record seeds for any randomized workload, input generator, or scheduler.
- Frequency scaling. Disable turbo/DVFS or report that it is on; otherwise speedups drift with
temperature and neighbors.
- Isolation. Run on a quiescent machine with no co-tenants; note it.
- Repeats. Multiple runs with reported variance — reproducibility includes the spread, not just
the mean.
Reproduce the trend, not just the number
Because evaluators have different hardware, design the package so the conclusion survives a
change of machine:
- Provide a scaled-down configuration whose curve shape (linear region, saturation point)
matches the paper, runnable on modest hardware.
- State the minimum configuration on which the claim still holds, and the tolerance within
which you consider a result reproduced (e.g., "monotone speedup to the machine's core count," or
"within 10% at matched core counts").
- Separate machine-dependent absolute throughput from machine-independent claims (relative speedup,
scaling shape, correctness).
Data and workloads
- Ship or point to the exact inputs (graphs, matrices, traces) with sizes and provenance, not
just a generator script — or ship the generator with its seed.
- For real-application workloads, document versions and how they were built; a benchmark suite name
without a version pin is not reproducible.
Correctness reproducibility
- If a model checker or race detector supports your correctness claim, ship its configuration and
the harness so the check itself can be re-run, not just described.
Pre-submission reproducibility pass
[Fresh checkout] build from a clean clone in the pinned container -> succeeds?
[Blind re-run] a colleague re-runs a scaling figure on a *different* machine -> trend matches?
[Provenance] every number in the paper traces to a script + a recorded machine config?
[Anonymity] the review-time package hides machine names, accounts, and personal repos?
Output format
[Environment] CPU/GPU/memory/interconnect/software/runtime fully stated? gaps: <...>
[Non-determinism] pinning, warm-up, seeds, frequency, isolation, repeats controlled? list gaps
[Trend portability] scaled config + minimum config + tolerance stated? yes/no
[Data] exact inputs or seeded generator shipped/pinned? yes/no
[Pass result] fresh-checkout build + cross-machine trend match? yes/no
1---2name: ppopp-reproducibility3description: Use when making a PPoPP paper's parallel-performance results reproducible, covering the hardware and topology description reviewers re-run, thread pinning and NUMA control, seeds and warm-up, compiler/driver/flag provenance, and building an environment that reproduces the paper's scaling trend on different machines.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/ppopp-reproducibility6---78# PPoPP Reproducibility910Build the reproducibility story that a parallel-performance paper needs. Unlike a deterministic11algorithm, a PPoPP result depends on the **machine, the topology, and the run conditions** — and an12evaluator or a future reader will not have your exact node. Reproducibility here means someone else13can rebuild the environment and recover the **scaling trend and the relative comparison**, even when14their absolute numbers differ. Pin everything you can at run time; you cannot reconstruct it after15the machine is released.1617## The environment description reviewers actually re-run1819State enough that a stranger could stand up the same experiment:2021```text22[CPU] exact model, sockets, cores/threads, base/turbo policy, NUMA node layout23[Memory] size, channels, speed; per-socket bandwidth if it bounds you24[GPU] model, count, driver + CUDA/ROCm version, connection (PCIe/NVLink)25[Interconnect] for multi-node: fabric and topology26[Software] OS + kernel, compiler + version + exact flags, libraries + versions, allocator27[Runtime] thread count(s), pinning/affinity policy, scheduler settings, env vars28```2930An evaluation whose machine is described only as "a Linux server" is the one reviewers trust least31and cannot reproduce.3233## Control the sources of parallel non-determinism3435- **Thread pinning / affinity.** Pin threads to cores and state the policy; unpinned runs migrate36 and produce unrepeatable numbers. Document NUMA placement (e.g., `numactl`, first-touch policy).37- **Warm-up.** Discard JIT, cache, and allocator warm-up; report the steady-state measurement38 protocol and how many iterations you drop.39- **Seeds.** Fix and record seeds for any randomized workload, input generator, or scheduler.40- **Frequency scaling.** Disable turbo/DVFS or report that it is on; otherwise speedups drift with41 temperature and neighbors.42- **Isolation.** Run on a quiescent machine with no co-tenants; note it.43- **Repeats.** Multiple runs with reported variance — reproducibility includes the spread, not just44 the mean.4546## Reproduce the trend, not just the number4748Because evaluators have different hardware, design the package so the **conclusion** survives a49change of machine:5051- Provide a **scaled-down configuration** whose curve *shape* (linear region, saturation point)52 matches the paper, runnable on modest hardware.53- State the **minimum configuration** on which the claim still holds, and the **tolerance** within54 which you consider a result reproduced (e.g., "monotone speedup to the machine's core count," or55 "within 10% at matched core counts").56- Separate machine-dependent absolute throughput from machine-independent claims (relative speedup,57 scaling shape, correctness).5859## Data and workloads6061- Ship or point to the **exact inputs** (graphs, matrices, traces) with sizes and provenance, not62 just a generator script — or ship the generator *with* its seed.63- For real-application workloads, document versions and how they were built; a benchmark suite name64 without a version pin is not reproducible.6566## Correctness reproducibility6768- If a model checker or race detector supports your correctness claim, ship its configuration and69 the harness so the check itself can be re-run, not just described.7071## Pre-submission reproducibility pass7273```text74[Fresh checkout] build from a clean clone in the pinned container -> succeeds?75[Blind re-run] a colleague re-runs a scaling figure on a *different* machine -> trend matches?76[Provenance] every number in the paper traces to a script + a recorded machine config?77[Anonymity] the review-time package hides machine names, accounts, and personal repos?78```7980## Output format8182```text83[Environment] CPU/GPU/memory/interconnect/software/runtime fully stated? gaps: <...>84[Non-determinism] pinning, warm-up, seeds, frequency, isolation, repeats controlled? list gaps85[Trend portability] scaled config + minimum config + tolerance stated? yes/no86[Data] exact inputs or seeded generator shipped/pinned? yes/no87[Pass result] fresh-checkout build + cross-machine trend match? yes/no88```