brycewang-stanford VIS Artifact Evaluation
Use this for the reproducibility track. IEEE VIS does not use ACM-style artifact badges; its recognition is the Graphics Replicability Stamp Initiative (GRSI) — issued to TVCG papers as the TVCG Replicability Stamp — plus the conference's Open Practices program. Two things to internalize: the stamp is earned by an independent volunteer actually reproducing your results from a public archive, and the review artifact (anonymized, for your paper's reviewers) is not the same deliverable as the stamp artifact (de-anonymized, permanently archived).
What GRSI certifies (verify the current process)
| Mechanism | What it certifies | What earns it |
|---|---|---|
| TVCG Replicability Stamp (GRSI) | An independent volunteer reproduced the paper's results from your code/data | A public repository + a single documented build/run path that regenerates the key figures/results |
| Open Practices disclosures | Transparent reporting of open code, data, preprints, preregistration | Filling the camera-ready Open Practices form honestly and posting the materials |
Unlike a graded badge ladder, the stamp is binary: an evaluator either reproduces your results or does not. The failure mode is therefore always "it did not build/run on their machine," never "the idea was weak" — design for a stranger's clean environment.
What a GRSI volunteer opens first
| Claim type | First thing reproduced | Common failure caught |
|---|---|---|
| A visualization technique/algorithm | The build + a script that regenerates a key figure | Undocumented deps; only-builds-on-authors'-GPU |
| A system/tool | The install and a demo on bundled sample data | Requires a private server, API key, or paid license |
| A perceptual/empirical study | The analysis scripts that turn raw responses into the paper's stats | Numbers in the PDF no script reproduces; raw data missing |
| A rendering result | The pipeline + reference images with a comparison | Non-deterministic output with no tolerance/seed documented |
Assume the evaluator gives your package a bounded time budget on a clean machine. The first build and the first regenerated figure must succeed.
Packaging plan
[Container] ship a Dockerfile or a pinned environment (requirements/lockfile, exact toolchain
versions); avoid "install these 40 things by hand" and undocumented GPU/driver needs
[README] one-screen orientation: what it is, how to build, how to run the demo, how to
regenerate each figure/result, expected runtime and outputs
[Mapping] an explicit table: paper figure/result -> script -> expected output
[Data] the actual dataset or stimuli (or documented access), not just a pointer
[Determinism] seeds, tolerances, and reference images for anything stochastic or GPU-dependent
[License] an OSI-approved license so others can reuse the visualization code
[Archive] deposit in a DOI-issuing repository (OSF, Zenodo, Software Heritage) for permanence
Anonymized review artifact vs. stamp artifact
- At submission (if double-blind): the supplemental code/data/video is anonymized for the paper's reviewers — no owner strings, lab names, institutional URLs, or identity-revealing demo links, and no live repository that discloses authors.
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version a GRSI volunteer reproduces and the camera-ready cites.
Worked vignette: stamping a technique + system paper
A paper contributes a new graph-layout technique and an interactive system. To target the stamp:
ship a Docker image with the layout code pre-built; a run_demo.sh that lays out a small bundled
graph and writes the teaser figure in under a minute; a reproduce/ directory whose scripts
regenerate each quantitative figure from logged benchmark data; a figure-to-script mapping in the
README; the benchmark graphs themselves with provenance; and an MIT/BSD license. State honestly
which figures are turnkey and which need the full (slow) benchmark run.
Calibration
- The GRSI review is independent of and after camera-ready; do not conflate them, and do not block the paper on the stamp result.
- The exact GRSI submission process, the Open Practices requirements, and whether any element is mandatory vary by cycle — confirm on the current Open Practices page and the GRSI site.
Output format
[Target recognition] TVCG Replicability Stamp / Open Practices disclosures
[Artifact role] anonymized review artifact / public stamp artifact
[Contents] <code/data/stimuli/determinism aids/license>
[Clean-machine test] does build + demo + one regenerated figure succeed? yes/no
[Figure mapping] <figure/result -> script -> expected output present? yes/no>
[Fixes before archiving] <ordered list>