VIS Reproducibility
Use this before submission and again before camera-ready. IEEE VIS's Open Practices posture and the
Graphics Replicability Stamp make reproducibility a visible dimension, not a courtesy: reviewers
routinely open the supplemental code, data, and video, and the TVCG camera-ready collects
open-practices disclosures. The goal is that a competent reader could rebuild your figures, rerun
your study analysis, and reach your conclusions.
Evidence map
- Map each figure, quantitative result, and study finding to a verifiable location — a
section, a figure generated from logged data, or a script in the supplemental archive.
- For techniques and rendering, give enough of the algorithm, parameters, and environment
(including GPU/driver assumptions and tolerances) that a reader could re-implement or re-run.
- For empirical and perceptual studies, report participants and recruitment, apparatus/stimuli,
the task, the design (within/between), measures, statistics, and the analysis scripts.
- Keep the open-materials statement truthful and specific: what is shared, where it lives, and —
if something cannot be shared — exactly why.
- Keep the paper and the archive consistent: a number in the PDF that no script reproduces is
the contradiction reviewers read as carelessness.
Open-materials statement audit
| Claim in the paper |
Weak availability answer |
VIS-ready answer |
| "We render/lay out X" |
"Code available on request" |
Public, licensed repo with a build path that regenerates the teaser figure |
| "Our system supports task Y" |
"Demo will be released" |
Runnable build (or Docker) with bundled sample data and a demo script |
| "N participants judged Z" |
Nothing (privacy cited vaguely) |
Anonymized responses, stimuli, the analysis notebook, and the ethics/consent note |
| "We evaluated on dataset D" |
Named but not shared |
The dataset or documented access + the preprocessing scripts |
"Available on request" reads as not available; convert every such line into a concrete,
anonymized archive or an explicit, justified exception.
Preregistration for studies (a distinctly VIS-valued move)
Perceptual experiments and controlled user studies benefit from preregistration (e.g., on OSF):
locking hypotheses, design, sample size, and the analysis plan before data collection separates
confirmatory from exploratory findings and blunts the "you fished for that result" objection.
[Preregister] hypotheses, conditions, planned N + power analysis, primary DV, analysis plan
[Cite it] reference the (anonymized) preregistration in the paper; report deviations honestly
[Separate] label confirmatory vs. exploratory results; do not present post-hoc as planned
Provenance pinning
[Datasets] record source and version; archive the actual data or stimuli, not just a query/URL;
document any cleaning/filtering with the script
[Rendering] pin toolchain and library versions; provide reference images and a comparison
tolerance for GPU-dependent or non-deterministic output
[Studies] store raw per-participant responses (anonymized), the exact stimuli, and timing
[Compute] state hardware and runtime so a reader can size a reproduction
[Randomness] log seeds; say what is and is not deterministic
Degrees of reproducibility (state the one you achieved)
- Turnkey: one documented command regenerates each figure/result from logged data.
- Scripted: scripts exist but need documented manual steps, large data, or specific hardware.
- Descriptive: prose detailed enough that a competent reader could rebuild the pipeline.
For VIS, aim turnkey for anything an evaluator might rerun quickly (a figure from logged
benchmark data, a study's statistics from anonymized responses); a large rendering benchmark or a
proprietary dataset may stay scripted with access clearly documented. Stating the achieved level
honestly beats promising turnkey behavior that fails on a clean machine — the GRSI stamp is decided
exactly there.
Vignette: a technique-plus-study paper
A paper contributing a new encoding and a controlled study evaluating it. Its reproducibility
spine: the encoding code with a script that regenerates each figure; the study's stimuli and
anonymized per-participant responses; the preregistration for the confirmatory hypotheses; the
analysis notebook that turns responses into the reported effect sizes and CIs; and one honest
sentence about anything (identifiable video, proprietary data) that cannot be shared and why.
Consistency and camera-ready pass
- Before submission: every scored number and figure traces to the archive; the open-materials
statement matches reality; if double-blind, the archive is anonymized (no owner strings, lab
names, or institutional URLs).
- Before camera-ready: swap anonymized links for permanent, DOI-issuing archives, complete the Open
Practices form, and align the package with what you submit to GRSI (
vis-artifact-evaluation).
Output format
[Claim inventory] <figure/result/finding -> evidence location>
[Open materials] concrete / vague / missing
[Preregistration] present / not applicable / should have (for studies)
[Provenance gaps] <dataset versions / rendering references / study raw data / seeds>
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF>
[Archive fixes] <additions before upload>
1---2name: vis-reproducibility3description: Use when strengthening IEEE VIS reproducibility and open-practices evidence, covering the open-materials statement, anonymized-but-runnable code and stimuli, preregistration of perceptual and user studies, provenance for datasets and rendering pipelines, claim-to-figure mapping, honest degrees of reproducibility, and consistency between what the TVCG paper says and what the supplemental archive contains.4---56# VIS Reproducibility78Use this before submission and again before camera-ready. IEEE VIS's Open Practices posture and the9**Graphics Replicability Stamp** make reproducibility a visible dimension, not a courtesy: reviewers10routinely open the supplemental code, data, and video, and the TVCG camera-ready collects11open-practices disclosures. The goal is that a competent reader could rebuild your figures, rerun12your study analysis, and reach your conclusions.1314## Evidence map1516- Map each **figure, quantitative result, and study finding** to a **verifiable location** — a17 section, a figure generated from logged data, or a script in the supplemental archive.18- For **techniques and rendering**, give enough of the algorithm, parameters, and environment19 (including GPU/driver assumptions and tolerances) that a reader could re-implement or re-run.20- For **empirical and perceptual studies**, report participants and recruitment, apparatus/stimuli,21 the task, the design (within/between), measures, statistics, and the analysis scripts.22- Keep the **open-materials statement** truthful and specific: what is shared, where it lives, and —23 if something cannot be shared — exactly why.24- Keep the paper and the archive **consistent**: a number in the PDF that no script reproduces is25 the contradiction reviewers read as carelessness.2627## Open-materials statement audit2829| Claim in the paper | Weak availability answer | VIS-ready answer |30|---|---|---|31| "We render/lay out X" | "Code available on request" | Public, licensed repo with a build path that regenerates the teaser figure |32| "Our system supports task Y" | "Demo will be released" | Runnable build (or Docker) with bundled sample data and a demo script |33| "N participants judged Z" | Nothing (privacy cited vaguely) | Anonymized responses, stimuli, the analysis notebook, and the ethics/consent note |34| "We evaluated on dataset D" | Named but not shared | The dataset or documented access + the preprocessing scripts |3536"Available on request" reads as *not available*; convert every such line into a concrete,37anonymized archive or an explicit, justified exception.3839## Preregistration for studies (a distinctly VIS-valued move)4041Perceptual experiments and controlled user studies benefit from **preregistration** (e.g., on OSF):42locking hypotheses, design, sample size, and the analysis plan before data collection separates43confirmatory from exploratory findings and blunts the "you fished for that result" objection.4445```text46[Preregister] hypotheses, conditions, planned N + power analysis, primary DV, analysis plan47[Cite it] reference the (anonymized) preregistration in the paper; report deviations honestly48[Separate] label confirmatory vs. exploratory results; do not present post-hoc as planned49```5051## Provenance pinning5253```text54[Datasets] record source and version; archive the actual data or stimuli, not just a query/URL;55 document any cleaning/filtering with the script56[Rendering] pin toolchain and library versions; provide reference images and a comparison57 tolerance for GPU-dependent or non-deterministic output58[Studies] store raw per-participant responses (anonymized), the exact stimuli, and timing59[Compute] state hardware and runtime so a reader can size a reproduction60[Randomness] log seeds; say what is and is not deterministic61```6263## Degrees of reproducibility (state the one you achieved)6465- **Turnkey:** one documented command regenerates each figure/result from logged data.66- **Scripted:** scripts exist but need documented manual steps, large data, or specific hardware.67- **Descriptive:** prose detailed enough that a competent reader could rebuild the pipeline.6869For VIS, aim **turnkey** for anything an evaluator might rerun quickly (a figure from logged70benchmark data, a study's statistics from anonymized responses); a large rendering benchmark or a71proprietary dataset may stay scripted with access clearly documented. Stating the achieved level72honestly beats promising turnkey behavior that fails on a clean machine — the GRSI stamp is decided73exactly there.7475## Vignette: a technique-plus-study paper7677A paper contributing a new encoding and a controlled study evaluating it. Its reproducibility78spine: the encoding code with a script that regenerates each figure; the study's stimuli and79anonymized per-participant responses; the preregistration for the confirmatory hypotheses; the80analysis notebook that turns responses into the reported effect sizes and CIs; and one honest81sentence about anything (identifiable video, proprietary data) that cannot be shared and why.8283## Consistency and camera-ready pass8485- Before submission: every scored number and figure traces to the archive; the open-materials86 statement matches reality; if double-blind, the archive is anonymized (no owner strings, lab87 names, or institutional URLs).88- Before camera-ready: swap anonymized links for permanent, DOI-issuing archives, complete the Open89 Practices form, and align the package with what you submit to GRSI (`vis-artifact-evaluation`).9091## Output format9293```text94[Claim inventory] <figure/result/finding -> evidence location>95[Open materials] concrete / vague / missing96[Preregistration] present / not applicable / should have (for studies)97[Provenance gaps] <dataset versions / rendering references / study raw data / seeds>98[Reproducibility level] turnkey / scripted / descriptive, stated honestly99[Paper fixes] <must appear in the PDF>100[Archive fixes] <additions before upload>101```
brycewang-stanford