Fse Reproducibility

FSE Reproducibility

Use this before submission and again before camera-ready. FSE's open-science posture makes reproducibility a scored dimension, not a courtesy: the double-anonymous review already expects an inspectable artifact, and the PACMSE camera-ready expects a permanent one. The goal is that a competent reader could rebuild your evidence and reach your conclusions.

Evidence map

• Map each research-question answer, technique claim, and reported number to a verifiable location — a paper section, a table generated from logged data, or a script in the artifact.
• For techniques, give enough of the algorithm, parameters, and environment that a reader could re-implement or re-run it.
• For empirical studies, report subjects and their selection, data collection, preprocessing, metrics, statistics, and the analysis scripts.
• Keep the Data Availability statement truthful and specific: what is shared, where it will live after acceptance, and — if something cannot be shared — exactly why.
• Keep the paper and the artifact consistent: a number in the PDF that no script in the artifact produces is the contradiction reviewers read as carelessness.

Data Availability statement audit

"Available on request" is treated as not available at FSE; convert every such line into a concrete, anonymized artifact or an explicit, justified exception.

Provenance pinning

[Mining]   pin repository SHAs; record corpus extraction date; archive the extracted dataset,
           not just the query; document fork/duplicate/bot handling
[LLM]      record exact model identifiers + access dates; cache raw inputs and outputs; report
           sampling settings; prefer post-training-cutoff subjects to bound contamination
[Compute]  state hardware, runtime, and number of runs so a reader can size a reproduction
[Randomness] log seeds for any stochastic step; say what is and is not deterministic

Degrees of reproducibility (state the one you achieved)

• Turnkey: one documented command regenerates each table/figure from logged data.
• Scripted: scripts exist but require documented manual steps or external data access.
• Descriptive: prose detailed enough that a competent reader could rebuild the pipeline.

For FSE, aim turnkey for anything a reviewer might rerun quickly (a detection script on sample inputs, a plot from logged results); large mined corpora or industrial data may stay scripted with access clearly documented. Stating the achieved level honestly beats promising turnkey behavior that fails on a clean machine.

Vignette: a mixed-methods study

Consider a study combining mined pull-request data with a developer survey. Its reproducibility spine: the mining scripts with pinned SHAs and extraction date; the anonymized extracted dataset; the survey instrument and anonymized responses; the qualitative codebook with inter-rater agreement; and the analysis notebooks that turn all of it into the paper's tables — plus one honest sentence about the parts (raw identities, private repositories) that cannot be shared and why.

Consistency and camera-ready pass

• Before submission: every scored number traces to the artifact; the Data Availability statement matches reality; the artifact is anonymized (no owner strings, cluster paths, or lab names).
• Before camera-ready: swap anonymized links for permanent, DOI-issuing archives, and align the statement with the ACM artifact badges you are pursuing (fse-artifact-evaluation).

Output format

[Claim inventory] <claim -> evidence location>
[Data Availability] concrete / vague / missing
[Provenance gaps] <mining SHAs / LLM caching / seeds / compute>
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF>
[Artifact fixes] <additions before upload>

Source: brycewang-stanford/Awesome-Journal-Skills → FSE-Skills/skills/fse-reproducibility/SKILL.md