Vis Experiments

VIS Experiments

Use this before submission when the evaluation is not yet locked. IEEE VIS reviewers judge whether the evidence matches the contribution type — and visualization has several distinct contribution types, each with its own evidence standard. The organizing principle is evaluate the claim you actually make: a claim about perception needs a controlled study, a claim about scale needs a benchmark, a claim about real-world usefulness needs a design-study validation or a deployment.

Evaluation audit

• Pick the evaluation to the contribution type, not by habit (see the table). The classic VIS reject is a system paper "evaluated" only by an accuracy number, or a perceptual claim backed only by author intuition.
• Design controlled studies properly: state hypotheses, a within/between design, a task from a recognized task taxonomy, a power analysis justifying N, and report effect sizes with confidence intervals, not just p-values. Consider preregistration for confirmatory studies (vis-reproducibility).
• Justify encodings perceptually: color choices should be CVD-safe and appropriate to the data type (sequential/diverging/categorical); channel choices should follow known effectiveness rankings for the task. Reviewers check this explicitly.
• Benchmark techniques fairly: compare against the strongest existing technique and a reasonable baseline on realistic data sizes, with runtime/quality reported and the code available.
• Hold qualitative and design-study work to method: coding schemes, multiple coders, reflection across abstraction levels, and an audit trail — design studies are a first-class VIS contribution, not a weak substitute for a controlled study.
• Pin provenance for datasets, stimuli, and rendering so the evaluation reproduces rather than re-samples.

Contribution-type to evidence table

Controlled-study design floor

[Hypotheses]  stated before analysis; confirmatory vs. exploratory labeled
[Design]      within/between justified; counterbalancing; the task from a known taxonomy
[Power]       an a-priori power analysis justifies N; do not stop at "we recruited 20"
[Stimuli]     real or realistic; the exact stimuli archived
[Measures]    accuracy AND time AND (where relevant) preference/confidence; define each
[Statistics]  effect sizes + CIs; appropriate tests; corrections for multiple comparisons
[Reporting]   report what you found, including null and exploratory results, honestly

Perceptual and accessibility checks

• Color: use CVD-safe palettes; match palette type to data (sequential for ordered, diverging for a meaningful midpoint, categorical for nominal); never encode magnitude on hue alone.
• Channel effectiveness: prefer position/length for quantitative comparison; justify any use of area, angle, or color for a precise task.
• Legibility: ensure figures read in grayscale and at print size; a result a reviewer cannot see is a result you cannot claim.

Vignette: evaluating a new time-series encoding

Suppose the paper claims a new encoding reads trends faster than a line chart. The matching plan: a controlled within-subjects study; trend-reading tasks drawn from a task taxonomy; real time-series stimuli, archived; an a-priori power analysis fixing N; accuracy and completion-time as measures; effect sizes with CIs comparing the new encoding to a tuned line-chart baseline; a CVD-safe palette justified against the task; and honest reporting of any task where the line chart won — every number traceable to the archived analysis notebook.

Output format

[Evaluation readiness] strong / adequate / weak
[Contribution type] perceptual / technique / system / design-study / algorithm / data
[Evidence match] <contribution type -> evidence chosen -> appropriate? yes/no>
[Study rigor] <hypotheses? power analysis? effect sizes + CIs? preregistered?>
[Encoding validity] <CVD-safe? channel matched to task? grayscale-legible?>
[Provenance] <stimuli/data/rendering archived and reproducible? yes/no>
[Decision-critical next run] <one study or benchmark to add>

Source: brycewang-stanford/Awesome-Journal-Skills → VIS-Skills/skills/vis-experiments/SKILL.md