brycewang-stanford PLDI Experiments
A PLDI evaluation answers one question: does the claimed mechanism cause the
claimed effect on programs that matter? Everything in the design flows from
making that causal link auditable. The community's shared rubric is SIGPLAN's
Empirical Evaluation checklist (see pldi-reproducibility for the measurement
hygiene); this skill covers the design choices above the hygiene layer.
Benchmark choice is an argument, not a default
- Justify the suite relative to the claim: an allocation optimizer needs allocation-heavy programs and allocation-light ones (to show no regression); a parser-facing analysis needs real grammars, not microbenchmarks.
- Use community suites where they exist and state versions; add real-world applications when the suite is known to under-represent your phenomenon.
- List exclusions with reasons. "We exclude two SPEC programs that use
setjmp, which our restriction rejects (§4.4)" builds trust; silent dropping destroys it. - Include programs your technique should not help. Flat results on those are evidence the instrument works.
Baselines that survive the PC
| Weak move | Defensible move |
|---|---|
| Compare against -O0 or an untuned build | Strongest published configuration of the standard toolchain (state version + flags) |
| Reimplement a rival technique quickly | Use the authors' artifact where one exists; note deviations |
| Compare only against your own prior system | Add the external baseline reviewers will name in review |
| Report best-of-N runs | Report distribution over all N runs |
| One aggregate number | Aggregate + per-benchmark table, so wins and losses show |
The reviewers most likely to be assigned your paper wrote the baselines. Assume the baseline's author reads your flags line.
Ablations isolate the mechanism
The claim "our escape signatures cause the speedup" needs the experiment where signatures are replaced by the prior summary while everything else stays fixed. Design one ablation per mechanism named in the contributions list; a mechanism with no ablation is a mechanism the paper does not actually test.
The three currencies
Runtime, compile time, and memory are all first-class at PLDI. A technique that buys 1.1x runtime with 3x compile time must say so in the abstract, not in a footnote. Report all three, each with repetitions and dispersion, even when one of them is "no change" — especially when it is "no change."
Anticipated-objection pass
Run this list before the deadline; it is roughly what a PLDI review's evaluation section says when it goes badly:
[ ] Is the delta bigger than the noise band? (CI overlap check per benchmark)
[ ] Does the effect survive on a second microarchitecture?
[ ] Are the flags/version of every baseline stated and current?
[ ] Is there a benchmark where we lose, and do we explain it?
[ ] Does the ablation exist for every mechanism we claim credit for?
[ ] Is warmup/steady-state handling stated per benchmark family?
[ ] Could the speedup come from an unrelated engineering change? (same-codebase control)
Negative and neutral results
A paragraph explaining the two programs where the technique regresses — with a cause, not a shrug — routinely appears in accepted PLDI papers and in Distinguished Paper profiles. Reviewers read it as instrument calibration. Deleting the losing rows reads as the opposite.
Output format
[Suite] chosen + justified? versions pinned? exclusions listed?
[Baselines] strongest config? external baseline present? flags stated?
[Ablations] mechanism -> ablation experiment (n/n covered)
[Currencies] runtime / compile time / memory each measured with variance?
[Objection pass] items failing from the checklist above