brycewang-stanford PPoPP Artifact Evaluation
Use this for the artifact track. PPoPP runs a post-acceptance artifact evaluation with its own deadline (PPoPP 2026: artifact submission 17 November 2025, notification 5 January 2026), separate from the camera-ready and shared in culture with co-located CGO. Two things to internalize: PPoPP has a distinctive badge policy, and the hard part of a parallel artifact is that the results must reproduce on someone else's hardware, where core counts and topology differ from yours.
PPoPP's badge policy (verify the current set each cycle)
| Badge | Colour | What it certifies | How it is awarded |
|---|---|---|---|
| Artifacts Available | green | The artifact is publicly, permanently retrievable | By the publisher, from a deposited-artifact link — no formal audit |
| Artifacts Evaluated — Functional | lighter red | The artifact runs, is documented, consistent, complete, exercisable | AE committee runs it |
| Artifacts Evaluated — Reusable | darker red | Functional plus quality that exceeds minimal functionality; carefully documented and structured for reuse | AE committee runs it |
| Results Reproduced | darker blue | The paper's main results were obtained by the evaluators (within tolerance) | AE committee re-runs and matches |
Two PPoPP-specific facts to get right:
- The committee awards one of Functional / Reusable (Reusable is the higher bar) plus possibly Results Reproduced.
- PPoPP does not award the lighter-blue "Results Replicated" badge — do not target or claim it. Available is not evaluated; it is publisher-granted from a link, so it is the cheapest badge to secure and worth doing regardless.
What makes a parallel artifact hard
An evaluator does not have your machine. A speedup that depended on your 96-core dual-socket node will not reproduce on a reviewer's 16-core laptop, and "Results Reproduced" requires agreement within a stated tolerance. Design for portability of the conclusion, not the absolute numbers:
[Hardware doc] state the exact machine you used (CPU/GPU model, sockets, cores, NUMA, memory,
interconnect) and the minimum config on which the trend still holds
[Scaled runs] provide a small/fast configuration whose *shape* (linear region, saturation)
matches the paper, so an evaluator can reproduce the claim on modest hardware
[Tolerance] state explicitly what "reproduced" means for your numbers (e.g. within 10%, or
"monotone speedup to the machine's core count")
[Determinism] fix seeds; pin threads; document warm-up; make runs repeatable
[GPU] document driver/CUDA/ROCm versions and the GPU class needed; provide a CPU
fallback path where possible
What evaluators open first
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| A concurrent structure | README + a one-command throughput run | Only builds on the authors' toolchain; hardcoded core count |
| A runtime/scheduler | The script that reproduces a scaling figure | Numbers in the PDF that no script regenerates |
| A GPU technique | Build + a small kernel run | Requires a specific GPU with no fallback; missing driver versions |
| A parallel algorithm | Build + a scaled input run | Data missing; topology assumptions undocumented |
Assume a bounded time budget on a clean machine; make the first ten minutes succeed.
Packaging plan
[Container] a Dockerfile or pinned environment (spack/conda/lockfile); avoid manual 40-step builds
[README] one-screen orientation: what it is, how to build, how to run the demo, how to
reproduce each figure, expected runtime, and the hardware each run needs
[Mapping] an explicit table: paper claim/figure -> script -> expected result (+ tolerance)
[Scaling kit] scripts that sweep thread/core count and emit the paper's curve on the eval machine
[Provenance] exact hardware used, compiler/flags, driver versions, seeds, pinning policy
[License] an OSI-approved license so the artifact can be badged Reusable
[Archive] deposit in a DOI-issuing repository (Zenodo/figshare/Software Heritage) for Available
Anonymized review artifact vs. badge artifact
- At paper submission: the artifact (if attached for reviewers) is anonymized — no owner strings, cluster/account paths, lab names, or identifying links.
- At the AE track (post-acceptance): the de-anonymized, licensed, DOI-archived version is what evaluators badge and the camera-ready cites.
Calibration
- The AE deadline is after acceptance and independent of the camera-ready; do not conflate them.
- Confirm each cycle whether AE is run jointly with CGO, the exact badge set and colours, and whether Available is auto-granted by the publisher (as verified) or handled by the committee.
Output format
[Target badges] Available (publisher) / Functional or Reusable / Results Reproduced (NOT Replicated)
[Artifact role] anonymized review artifact / public badge artifact
[Portability] scaled config whose trend matches the paper on modest hardware? tolerance stated?
[Ten-minute test] build + demo succeeds on a clean machine? yes/no
[Claim mapping] each figure -> script -> expected result (+ tolerance) present? yes/no
[Fixes before upload] <ordered list>