FAccT Reproducibility
Use this before submission and again before camera-ready. At FAccT, transparency is not only the
subject of the field — it is a norm the community holds its own papers to. But FAccT
reproducibility is broader than "does the code run": it spans releasing and documenting the data
and models behind an audit, making a qualitative study auditable without exposing participants,
and being honest where confidentiality or proprietary access genuinely bars release. The goal is
that a competent reader could trace how you got from evidence to conclusion — and judge whether the
harm you claim is real.
Transparency map
- Map each finding to a verifiable location — a paper section, a table generated from released
analysis, a codebook, or a documented case record.
- For quantitative audits: release the analysis code, the dataset (or documented access), the
exact metrics and subgroup definitions, and the seeds/versions — enough that a reader could
re-run the disaggregation and reach your gaps.
- For qualitative/participatory work: release what can be shared safely — the interview
protocol, the codebook, aggregate coded results, consent materials — and state clearly what
cannot be shared and why (participant confidentiality, community agreement).
- Document datasets and models, not just release them. A datasheet for a dataset, a model
card for a model, and a data statement for a language corpus are the FAccT-native
documentation genres; use them to record provenance, composition, intended use, and known limits.
- Keep the paper and artifact consistent. A disparity in the PDF that no released analysis
reproduces is the contradiction reviewers read as carelessness — or worse, as an unfalsifiable
harm claim.
Documentation-and-availability audit
| Claim in the paper |
Weak availability answer |
FAccT-ready answer |
| "We audit N deployed systems" |
"Data available on request" |
Released dataset (or documented access) + analysis code + subgroup definitions |
| "Our dataset is representative" |
Raw files with no context |
A datasheet: how collected, who is in it, gaps, intended and off-label uses |
| "Our model behaves fairly" |
Weights only |
A model card: evaluation disaggregated by group, intended use, known failure groups |
| "We interviewed P affected people" |
Nothing (privacy cited vaguely) |
Protocol + codebook + aggregate results + a clear, specific confidentiality boundary |
| "The LLM produced these outputs" |
"We used a chatbot" |
Model IDs and dates, prompts, cached raw outputs, sampling settings |
"Available on request" reads as not available; convert every such line into a concrete release,
proper documentation, or an explicit, justified exception.
Provenance pinning
[Scraped/mined data] record source, extraction date, and terms; archive the extracted dataset,
not just the scraper; document deduplication and filtering
[Protected attributes] document how group labels were obtained/inferred and their error
[Models] record exact model identifiers + access dates; cache raw prompts and outputs;
report sampling settings; a live-API-only study re-samples, it does not reproduce
[Qualitative] version the codebook; log coding decisions; keep an audit trail a second
reader could follow
[Consent] keep the consent/ethics record aligned with what you release
Degrees of reproducibility (state the one you achieved)
- Turnkey: one documented command regenerates each disaggregated table/figure from released
data.
- Scripted: analysis scripts exist but need documented manual steps or restricted-data access.
- Documented: for qualitative or confidential work, the protocol, codebook, and aggregate
results let a reader audit the reasoning without re-running.
For FAccT, aim turnkey for anything a reviewer could rerun quickly (a fairness-metric recomputation,
a plot from released results); confidential interview data or proprietary system access stays
documented with the boundary stated. Stating the achieved level honestly beats promising turnkey
behavior that fails.
Vignette: a mixed-methods accountability study
Consider a study combining a quantitative audit of a benefits system with interviews of claimants.
Its transparency spine: the audit code with pinned data versions and subgroup definitions; the
released (or access-documented) audit dataset with a datasheet; the interview protocol, codebook,
and aggregate themes; the consent and ethics record; and one honest paragraph on what cannot be
shared (claimant identities, the agency's internal data) and why — so the audit is falsifiable and
the qualitative reasoning is auditable, without re-harming participants.
Consistency and camera-ready pass
- Before submission: every disparity/finding traces to released or documented evidence; datasheets
and model cards drafted; the artifact is anonymized (no author names, institution paths, or
identity-revealing repository).
- Before camera-ready: swap any anonymized link for a permanent one, finalize the datasheet/model
card, and align the Ethical Considerations and Adverse Impacts statements with what you release.
Output format
[Finding inventory] <finding -> evidence location>
[Availability] concrete release / documented access / vague / missing
[Documentation] <datasheet / model card / data statement present where relevant? yes/no>
[Provenance gaps] <scrape terms / proxy labels / model caching / codebook>
[Reproducibility level] turnkey / scripted / documented, stated honestly
[Paper fixes] <must appear in the PDF>
[Artifact fixes] <additions before upload, kept anonymous>
Source: brycewang-stanford/Awesome-Journal-Skills → FAccT-Skills/skills/facct-reproducibility/SKILL.md
1---2name: facct-reproducibility3description: Use when strengthening ACM FAccT transparency and reproducibility — releasing code, data, and analysis for quantitative audits; documenting datasets and models with datasheets, model cards, and data statements; making qualitative and participatory work auditable without breaking confidentiality; pinning provenance for scraped and model-generated data; and keeping the paper, the supplementary material, and any released artifact consistent.4---567# FAccT Reproducibility89Use this before submission and again before camera-ready. At FAccT, transparency is not only the10*subject* of the field — it is a norm the community holds its own papers to. But FAccT11reproducibility is broader than "does the code run": it spans **releasing and documenting** the data12and models behind an audit, making a **qualitative** study auditable without exposing participants,13and being honest where confidentiality or proprietary access genuinely bars release. The goal is14that a competent reader could trace how you got from evidence to conclusion — and judge whether the15harm you claim is real.1617## Transparency map1819- **Map each finding to a verifiable location** — a paper section, a table generated from released20 analysis, a codebook, or a documented case record.21- **For quantitative audits:** release the analysis code, the dataset (or documented access), the22 exact metrics and subgroup definitions, and the seeds/versions — enough that a reader could23 re-run the disaggregation and reach your gaps.24- **For qualitative/participatory work:** release what can be shared safely — the interview25 protocol, the codebook, aggregate coded results, consent materials — and state clearly what26 cannot be shared and why (participant confidentiality, community agreement).27- **Document datasets and models, not just release them.** A **datasheet** for a dataset, a **model28 card** for a model, and a **data statement** for a language corpus are the FAccT-native29 documentation genres; use them to record provenance, composition, intended use, and known limits.30- **Keep the paper and artifact consistent.** A disparity in the PDF that no released analysis31 reproduces is the contradiction reviewers read as carelessness — or worse, as an unfalsifiable32 harm claim.3334## Documentation-and-availability audit3536| Claim in the paper | Weak availability answer | FAccT-ready answer |37|---|---|---|38| "We audit N deployed systems" | "Data available on request" | Released dataset (or documented access) + analysis code + subgroup definitions |39| "Our dataset is representative" | Raw files with no context | A **datasheet**: how collected, who is in it, gaps, intended and off-label uses |40| "Our model behaves fairly" | Weights only | A **model card**: evaluation disaggregated by group, intended use, known failure groups |41| "We interviewed P affected people" | Nothing (privacy cited vaguely) | Protocol + codebook + aggregate results + a clear, specific confidentiality boundary |42| "The LLM produced these outputs" | "We used a chatbot" | Model IDs and dates, prompts, cached raw outputs, sampling settings |4344"Available on request" reads as *not available*; convert every such line into a concrete release,45proper documentation, or an explicit, justified exception.4647## Provenance pinning4849```text50[Scraped/mined data] record source, extraction date, and terms; archive the extracted dataset,51 not just the scraper; document deduplication and filtering52[Protected attributes] document how group labels were obtained/inferred and their error53[Models] record exact model identifiers + access dates; cache raw prompts and outputs;54 report sampling settings; a live-API-only study re-samples, it does not reproduce55[Qualitative] version the codebook; log coding decisions; keep an audit trail a second56 reader could follow57[Consent] keep the consent/ethics record aligned with what you release58```5960## Degrees of reproducibility (state the one you achieved)6162- **Turnkey:** one documented command regenerates each disaggregated table/figure from released63 data.64- **Scripted:** analysis scripts exist but need documented manual steps or restricted-data access.65- **Documented:** for qualitative or confidential work, the protocol, codebook, and aggregate66 results let a reader audit the reasoning without re-running.6768For FAccT, aim turnkey for anything a reviewer could rerun quickly (a fairness-metric recomputation,69a plot from released results); confidential interview data or proprietary system access stays70documented with the boundary stated. Stating the achieved level honestly beats promising turnkey71behavior that fails.7273## Vignette: a mixed-methods accountability study7475Consider a study combining a quantitative audit of a benefits system with interviews of claimants.76Its transparency spine: the audit code with pinned data versions and subgroup definitions; the77released (or access-documented) audit dataset with a datasheet; the interview protocol, codebook,78and aggregate themes; the consent and ethics record; and one honest paragraph on what cannot be79shared (claimant identities, the agency's internal data) and why — so the audit is falsifiable and80the qualitative reasoning is auditable, without re-harming participants.8182## Consistency and camera-ready pass8384- Before submission: every disparity/finding traces to released or documented evidence; datasheets85 and model cards drafted; the artifact is **anonymized** (no author names, institution paths, or86 identity-revealing repository).87- Before camera-ready: swap any anonymized link for a permanent one, finalize the datasheet/model88 card, and align the Ethical Considerations and Adverse Impacts statements with what you release.8990## Output format9192```text93[Finding inventory] <finding -> evidence location>94[Availability] concrete release / documented access / vague / missing95[Documentation] <datasheet / model card / data statement present where relevant? yes/no>96[Provenance gaps] <scrape terms / proxy labels / model caching / codebook>97[Reproducibility level] turnkey / scripted / documented, stated honestly98[Paper fixes] <must appear in the PDF>99[Artifact fixes] <additions before upload, kept anonymous>100```101102---103104**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `FAccT-Skills/skills/facct-reproducibility/SKILL.md`
thedixitjain