OOPSLA Reproducibility
OOPSLA carries a particular historical burden here: the venue itself
published the papers showing that sloppy runtime measurement produces wrong
conclusions — Georges, Buytaert & Eeckhout's statistical-rigor paper (OOPSLA
2007) and the DaCapo suite's methodology argument (OOPSLA 2006); see
resources/exemplars/library.md. Reviewers steeped in that lineage apply the
SIGPLAN Empirical Evaluation Guidelines
(sigplan.org/Resources/EmpiricalEvaluation/) as a working checklist, and
the two-round model gives them a Minor/Major Revision lever to demand rigor
rather than merely complain about it. Reproducibility work done before
Round N is cheaper than the revision it preempts.
The four guideline pillars, operationalized
| Pillar |
Reviewer question |
Concrete obligation in the paper |
| Clear claims |
What exactly is asserted, on what workloads, on what hardware? |
Claims scoped with population, platform, and configuration |
| Suitable comparison |
Is the baseline the strongest sensible one, correctly configured? |
Baseline versions, flags, and tuning documented |
| Principled benchmarks |
Why these programs/corpora and not cherry-picked ones? |
Selection rule stated; exclusions listed with reasons |
| Adequate data analysis |
Do the numbers separate signal from noise? |
Repetitions, warmup policy, dispersion, and summary statistic all named |
Managed-runtime and PL-specific traps
- JIT warmup: steady-state and startup are different claims; measure and
label both or pick one explicitly.
- Nondeterministic compilation: JIT tiering, GC scheduling, and ASLR mean
run-to-run variance is structural — report distributions, not best-of.
- Geometric vs arithmetic means across benchmarks: choose deliberately
and say why; ratios of means and means of ratios diverge.
- Corpus studies (the Meyerovich–Rabkin lane): repository selection bias,
fork/duplicate contamination, and time-of-scrape all belong in the paper,
since the corpus is the instrument.
- Mechanized proofs: state the proof assistant version, axioms/assumed
lemmas, and which theorems are checked vs paper-only.
Reproducibility ledger
Keep one machine-readable ledger from the first experiment; it becomes the
artifact's spine and the Data-Availability Statement's evidence.
experiment: table3-throughput
runtime: OpenJDK 21.0.2 (Temurin), -Xmx16g, JIT default
hardware: 2x Xeon 6338, 256 GiB, SMT off, governor=performance
benchmarks: dacapo-23.11-chopin subset (selection rule: R1)
protocol: 30 invocations x 10 iterations, discard warmup by CUSUM
stats: geomean ratio + 95% bootstrap CI, per-benchmark violin in appendix
seed_policy: fixed seeds logged; randomized order per invocation
data: raw CSV -> artifact path /results/table3/
Statement discipline
The Data-Availability Statement (required before the references —
oopsla-submission) is a promissory note the artifact must later redeem
under badge review (oopsla-artifact-evaluation). Write it from the ledger:
name what is included, what is excluded and why (license, privacy, scale),
and on what hardware results were produced. A statement that overpromises is
worse than a modest one — evaluators check.
Pre-round self-audit
- Re-derive every headline number from the ledger with one command.
- Delete one machine from the picture: does any claim silently depend on
unstated hardware?
- Hand a labmate the guidelines' four pillars and the PDF; each pillar they
cannot check off in the text is a revision demand waiting to be written.
Output format
[Pillar audit] claims/comparison/benchmarks/analysis: pass|gap each
[Runtime traps] <warmup, variance, mean-choice, corpus, proofs — issues found>
[Ledger] complete / missing fields: <list>
[Statement] redeemable as written: yes / overpromises: <items>
[Revision exposure] what a reviewer could demand in Round N+1
Source: brycewang-stanford/Awesome-Journal-Skills → OOPSLA-Skills/skills/oopsla-reproducibility/SKILL.md
1---2name: oopsla-reproducibility3description: Use when hardening an OOPSLA paper's empirical claims to the SIGPLAN Empirical Evaluation Guidelines — managed-runtime measurement discipline, warmup and variance reporting, corpus and benchmark provenance, environment pinning, and a Data-Availability Statement that the eventual artifact can actually honor.4---567# OOPSLA Reproducibility89OOPSLA carries a particular historical burden here: the venue itself10published the papers showing that sloppy runtime measurement produces wrong11conclusions — Georges, Buytaert & Eeckhout's statistical-rigor paper (OOPSLA122007) and the DaCapo suite's methodology argument (OOPSLA 2006); see13`resources/exemplars/library.md`. Reviewers steeped in that lineage apply the14SIGPLAN Empirical Evaluation Guidelines15(`sigplan.org/Resources/EmpiricalEvaluation/`) as a working checklist, and16the two-round model gives them a Minor/Major Revision lever to demand rigor17rather than merely complain about it. Reproducibility work done before18Round N is cheaper than the revision it preempts.1920## The four guideline pillars, operationalized2122| Pillar | Reviewer question | Concrete obligation in the paper |23| --- | --- | --- |24| Clear claims | What exactly is asserted, on what workloads, on what hardware? | Claims scoped with population, platform, and configuration |25| Suitable comparison | Is the baseline the strongest sensible one, correctly configured? | Baseline versions, flags, and tuning documented |26| Principled benchmarks | Why these programs/corpora and not cherry-picked ones? | Selection rule stated; exclusions listed with reasons |27| Adequate data analysis | Do the numbers separate signal from noise? | Repetitions, warmup policy, dispersion, and summary statistic all named |2829## Managed-runtime and PL-specific traps3031- **JIT warmup**: steady-state and startup are different claims; measure and32 label both or pick one explicitly.33- **Nondeterministic compilation**: JIT tiering, GC scheduling, and ASLR mean34 run-to-run variance is structural — report distributions, not best-of.35- **Geometric vs arithmetic means** across benchmarks: choose deliberately36 and say why; ratios of means and means of ratios diverge.37- **Corpus studies** (the Meyerovich–Rabkin lane): repository selection bias,38 fork/duplicate contamination, and time-of-scrape all belong in the paper,39 since the corpus *is* the instrument.40- **Mechanized proofs**: state the proof assistant version, axioms/assumed41 lemmas, and which theorems are checked vs paper-only.4243## Reproducibility ledger4445Keep one machine-readable ledger from the first experiment; it becomes the46artifact's spine and the Data-Availability Statement's evidence.4748```yaml49experiment: table3-throughput50runtime: OpenJDK 21.0.2 (Temurin), -Xmx16g, JIT default51hardware: 2x Xeon 6338, 256 GiB, SMT off, governor=performance52benchmarks: dacapo-23.11-chopin subset (selection rule: R1)53protocol: 30 invocations x 10 iterations, discard warmup by CUSUM54stats: geomean ratio + 95% bootstrap CI, per-benchmark violin in appendix55seed_policy: fixed seeds logged; randomized order per invocation56data: raw CSV -> artifact path /results/table3/57```5859## Statement discipline6061The Data-Availability Statement (required before the references —62`oopsla-submission`) is a promissory note the artifact must later redeem63under badge review (`oopsla-artifact-evaluation`). Write it from the ledger:64name what is included, what is excluded and why (license, privacy, scale),65and on what hardware results were produced. A statement that overpromises is66worse than a modest one — evaluators check.6768## Pre-round self-audit69701. Re-derive every headline number from the ledger with one command.712. Delete one machine from the picture: does any claim silently depend on72 unstated hardware?733. Hand a labmate the guidelines' four pillars and the PDF; each pillar they74 cannot check off in the text is a revision demand waiting to be written.7576## Output format7778```text79[Pillar audit] claims/comparison/benchmarks/analysis: pass|gap each80[Runtime traps] <warmup, variance, mean-choice, corpus, proofs — issues found>81[Ledger] complete / missing fields: <list>82[Statement] redeemable as written: yes / overpromises: <items>83[Revision exposure] what a reviewer could demand in Round N+184```8586---8788**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `OOPSLA-Skills/skills/oopsla-reproducibility/SKILL.md`
thedixitjain