brycewang-stanford SIGGRAPH Artifact Evaluation (Replicability Stamp)
SIGGRAPH does not run the ACM Artifact Review and Badging scheme that SIGSOFT and systems
venues use. The computer-graphics community's equivalent is a community-run replicability
stamp, earned after acceptance and independent of the review: the Graphics Replicability
Stamp Initiative (GRSI, replicabilitystamp.org) and the related Code Replicability in Computer
Graphics (CRCG, replicability.graphics). The distinction matters: a stamp certifies that a
volunteer rebuilt your code and reproduced your paper's results, and archives that code for the
community. Facts below trace to resources/official-source-map.md; confirm the current GRSI
process before you package.
What the stamp is (and is not)
| Graphics Replicability Stamp (GRSI) | ACM Artifact Badges (other venues) | |
|---|---|---|
| Who runs it | Volunteer researchers from the graphics community | The venue's artifact evaluation committee |
| What it certifies | Your code replicates the paper's results | Available / Functional / Reusable / Reproduced |
| When | Post-acceptance, on the initiative's own schedule | Post-acceptance, on the venue's AE deadline |
| Archiving | Software Heritage snapshot + long-term ID (since 2023) | DOI-issuing archive |
| Scope | Independent of SIGGRAPH's review; optional recognition | Tied to the venue's AE track |
The GRSI has recognized graphics code since 2016 and is supported across the field's venues (ACM TOG, IEEE TVCG, Wiley CGF, Elsevier C&G, CAD/CAGD). CRCG has, since July 2020, focused specifically on checking whether SIGGRAPH papers' results are replicable. Neither is required for publication — but the stamp is the credible, checkable signal of a runnable contribution in this community.
What a volunteer actually does
Assume a graphics-literate volunteer clones your repository on their own machine and tries to regenerate a headline result from your paper. They are checking replicability of results, not just that the code compiles:
| Contribution type | What the volunteer reproduces | Common failure caught |
|---|---|---|
| A rendering technique | A converged image matching a paper figure, within tolerance | Non-deterministic output; missing scene assets |
| A geometry/mesh method | A processed mesh matching the reported statistics | Hard-coded absolute paths; unshipped input meshes |
| A simulation | A representative frame/sequence from the paper | Unseeded RNG; platform-specific solver drift |
| A learning-based method | A result from released weights, not retraining from scratch | Weights absent; inference needs undocumented data |
Design so the first headline result reproduces from a documented command on a clean checkout.
Packaging plan for a replicable graphics artifact
[Build] a documented, pinned build (CMake/conda/Docker) that compiles on a clean machine;
name exact compiler/CUDA/driver versions where GPU code is involved
[Assets] ship (or give a stable download for) the scenes, meshes, textures, and weights the
results need -- code without inputs cannot replicate a figure
[Determinism] fix seeds; document tolerance for floating-point/GPU non-determinism; state which
results are bit-exact vs perceptually-equal
[Mapping] a table: paper figure/table -> command -> expected output image/metric/frame
[Timings] report the hardware and the wall-clock the paper claims, so timings are checkable
[License] an OSI-approved license so the code can be shared and reused
[Archive] a Software Heritage snapshot (GRSI archives accepted code there) + a Zenodo DOI
Determinism is the graphics-specific hard part
Reproducing an image is not reproducing a number. Plan for it:
- State the comparison metric and tolerance. Volunteers cannot judge "looks right" — give PSNR/SSIM/LPIPS or a diff threshold against a bundled reference image.
- Pin the stochastic pieces. Monte Carlo renderers, stochastic simulations, and GPU reductions drift; seed them and document the residual non-determinism.
- Bundle reference outputs. Include the exact images/frames the paper reports so a volunteer compares against ground truth, not a re-render of their own.
- Separate "replicate the result" from "retrain the model." For learning-based work, ship weights and an inference path; full retraining is rarely the replication target.
Calibration
- The stamp is post-acceptance and optional; it is decided by the initiative's volunteers on their timeline, not by the SIGGRAPH committee, and not on the camera-ready deadline.
- GRSI and CRCG are related but distinct efforts; check each initiative's current submission instructions and supported-venue list before submitting your code.
- A stamp strengthens a paper's reach but is never a substitute for in-paper evidence — the review already happened without it.
Output format
[Target] Graphics Replicability Stamp (GRSI) / CRCG check / both
[Headline result] <figure/table the volunteer will reproduce>
[Clean-machine build] compiles + runs from documented command? yes/no
[Assets] scenes/meshes/weights shipped or stably linked? yes/no
[Determinism] seeds fixed + tolerance + reference outputs bundled? yes/no
[Claim mapping] <figure -> command -> expected output present?>
[Archive] Software Heritage snapshot + DOI + OSI license? yes/no
[Fixes before submitting code] <ordered>