brycewang-stanford CSCW Reproducibility and Transparency
Reproducibility at CSCW cannot mean "rerun my script, get my table" — most of the venue's evidence is people, and much of it must never leave the research team. The venue's real standard is auditability: a skeptical reader should be able to see how you got from data to claims, and to build on the work, even where they cannot re-execute it. Different strands of a paper owe different transparency debts.
What each strand owes
| Evidence strand | Shareable | Auditable instead of shareable |
|---|---|---|
| Interviews / fieldwork | Interview guide, recruitment text, codebook with definitions and example (paraphrased) excerpts | The analysis trail: coding approach, memo practice, how disagreements were resolved, how themes stabilized |
| Trace / log analysis | Pipeline code, query definitions, aggregated datasets, synthetic samples | Exact API/version/date of collection; filtering decisions with counts at each step; bot/deletion handling |
| Surveys | Full instrument, scale provenance, analysis scripts | Sampling frame, response/nonresponse accounting |
| Deployments | System code or architecture description, condition assignment logic | Site-selection reasoning; what the deployment context makes non-portable |
| Statistics anywhere | Analysis scripts keyed to each table/figure | Pre-specification vs. exploration, stated honestly |
The qualitative transparency trail
You cannot share transcripts; you can share how you thought. The auditable minimum for interpretive work:
- A codebook that could be picked up by a stranger — code names, definitions, inclusion/exclusion notes, and one paraphrased exemplar each. Whether inter-rater statistics belong depends on the tradition; saying which tradition and why is the transparency act.
- A decision log of analytic turning points: when categories merged, what disconfirming cases forced revisions. Two paragraphs in an appendix outperform a ritual "themes emerged."
- Quote provenance discipline: every quotation traceable (internally) to a participant and context, with the paraphrase/alteration policy stated in the paper.
The trace-pipeline ledger
Platform data rots. Reviewers and future researchers need the ledger even when the data cannot travel:
[Source] platform, endpoint/API version, collection dates
[Scope] query terms / community list / time window, with the WHY
[Attrition] rows at each filter step: raw → deduplicated → bot-filtered →
analysis set (counts, not adjectives)
[Constructs] each analysis variable → the raw field(s) it derives from →
the practice it is claimed to measure
[Fragility] what breaks if the platform changes (API terms, deletion policy)
[Release] what is shared: code / aggregates / synthetic sample / nothing + reason
Honest availability statements
Write the data statement as a truth-telling exercise, not boilerplate. Three honest shapes:
- "Analysis code and aggregated measures are available at ; raw traces cannot be redistributed under the platform's terms and our ethics protocol."
- "The codebook, interview guide, and consent materials are provided; transcripts are not shareable under the consent participants gave — we chose consent terms that protected candor over shareability, and say so."
- "A synthetic dataset preserving the marginal distributions is provided for pipeline verification."
What never survives review twice (remember the same reviewers return at R&R): "data available upon reasonable request" with no request path, and claims of sharing that the supplement does not actually contain.
Preregistration and its limits
For confirmatory quantitative strands, preregistration strengthens the paper — link it anonymized (registries support anonymous view links). Do not force exploratory or interpretive work into a preregistration costume; labeling exploration honestly is the venue's norm.
Transparency audit
[Per strand] shareable artifacts listed and actually present? y/n
[Qualitative] codebook + decision log exist? tradition named? y/n
[Trace] ledger complete incl. attrition counts? y/n
[Statement] availability text matches reality exactly? y/n
[Ethics gate] every shared artifact re-checked against consent scope? y/n
Run the gate last and strictly: a transparency package that violates a consent
agreement is not a reproducibility win, it is a research-ethics failure that
cscw-artifact-evaluation exists to prevent.