brycewang-stanford PLDI Reproducibility
PLDI's methodological yardstick is written down: the SIGPLAN Empirical Evaluation
Guidelines and their one-page checklist (Blackburn, Hauswirth, Berger, Hicks,
Krishnamurthi, 2018; sigplan.org/Resources/EmpiricalEvaluation/, read 2026-07-08).
Reviewers and artifact evaluators both reach for it. This skill turns the
checklist into compiler-bench practice; pldi-experiments covers what to measure,
this covers whether anyone can trust and repeat the measurement.
Checklist, translated to PL systems
| Guideline item | What it means for a compiler/runtime paper |
|---|---|
| Clearly stated claims | "1.17x geomean on suite S vs baseline B at -O2" — never "significant speedups" |
| Suitable comparison | The strongest sensible baseline configuration, tuned as its authors intend |
| Principled benchmark choice | The suite is justified; exclusions are listed with reasons, not silently dropped |
| Adequate data analysis | Repetitions, variance, and an aggregation rule (geomean for ratios) stated in the paper |
The measurement sins PLDI reviewers hunt
- No warmup discipline. JIT-compiled and cache-sensitive workloads need documented warmup iterations before timed runs; AOT binaries still need file-cache and frequency-scaling control. Say which regime you measured — steady-state and cold-start are different claims.
- Single-run numbers. Report repetitions (dozens, not three), dispersion (confidence intervals or at least min/max), and never present a 2% delta inside the noise band as an improvement.
- One machine, universal claim. A locality optimization can invert between microarchitectures. Two platforms with differing cache hierarchies is the floor for a general performance claim; otherwise scope the claim to the measured machine.
- Unpinned toolchains. "GCC" is not a baseline; "GCC 14.2, -O2, glibc 2.39, Ubuntu 24.04, governor=performance" is.
- Benchmark survivorship. Excluding the programs your technique fails on, without saying so, is the most damaging silent choice in a PL evaluation.
A protocol worth writing down
Keep the protocol in the repository, executed by machine, so paper and artifact cannot diverge:
# protocol.sh — executed, not described
set -euo pipefail
lscpu > results/env/cpu.txt; uname -a > results/env/os.txt
cc --version > results/env/toolchain.txt
for b in $(cat benchmarks/suite.list); do
for i in $(seq 1 5); do ./run.sh "$b" >/dev/null; done # warmup
for i in $(seq 1 30); do ./run.sh "$b" >> "results/raw/$b.csv"; done
done
python3 scripts/aggregate.py --stat geomean --ci 95 results/raw/
Log the environment beside the numbers: CPU model, frequency-scaling governor, ASLR setting, load conditions. When a reviewer's rerun differs from yours, the environment log is what turns a dispute into a diagnosis.
Compile-time and memory are claims too
If the paper claims low compile-time overhead or memory neutrality, those numbers need the same repetitions-and-variance treatment as speedups. A "under 3% overhead" sentence backed by one timed build is the soft spot response-phase reviewers press hardest.
Tie-in to badges
Everything above lands in the artifact (pldi-artifact-evaluation): the executed
protocol becomes reproduce_all.sh, the environment log becomes results/env/,
and the suite-choice justification becomes benchmarks/README. Reproducibility
retrofitted after acceptance always shows.
Output format
[Guidelines pass] claims / comparison / benchmark choice / analysis — each ok?
[Warmup regime] documented? steady-state vs cold-start stated?
[Variance] runs per data point, CI method, noise floor vs claimed delta
[Platforms] n machines; claim scoped accordingly?
[Pinning + log] toolchain versions, flags, environment captured in repo?