FAccT Artifact Evaluation
Use this for the material that backs a FAccT paper's transparency and accountability claims. Note
the venue difference up front: FAccT does not run the SIGSOFT-style ACM Artifact Review and
Badging track that software-engineering venues use, and it does not hand out Available/Functional/
Reusable/Reproduced badges. 待核实: confirm on the current Author Guide whether any optional
artifact/reproducibility appendix or badge scheme has been added for your cycle. What FAccT does
have is a strong norm of accountability documentation — datasheets, model cards, data
statements, audit trails, and impact assessments — plus released code and data. Treat those genres
as your artifact and make each one credible on its own.
The FAccT documentation genres (know which your paper needs)
| Genre |
What it documents |
When your paper needs it |
| Datasheet for a dataset |
Motivation, composition, collection, preprocessing, uses, distribution, maintenance |
You release or rely on a dataset |
| Model card |
Intended use, training data, evaluation disaggregated by group, ethical considerations, limits |
You release or audit a model |
| Data statement (for language data) |
Speaker/annotator demographics, curation rationale, language variety |
You build or study a text/NLP corpus |
| Audit / evaluation report |
Method, subgroup metrics, thresholds, what was and was not tested |
Your contribution is an audit |
| Impact / risk assessment |
Foreseeable harms, affected populations, mitigations, residual risk |
Deployment or dual-use is plausible |
Pick the genres your claims actually require; a model audit with no model card, or a dataset paper
with no datasheet, reads as incomplete to this community.
What a credible documentation artifact contains
[Provenance] where the data/model came from, when, under what terms and consent
[Composition] who/what is in it, who is absent, and the resulting blind spots
[Disaggregation] evaluation broken out by protected/affected subgroup, with uncertainty
[Intended use] what it is for — and an explicit "off-label" / do-not-use list
[Limits & harms] known failure groups and foreseeable adverse impacts, not just accuracy
[Maintenance] who updates it, how issues are reported, how long it persists
[License] a clear license for released code/data so others can lawfully reuse it
Released code and data (the reproducibility half)
- Ship the analysis that turns data into the paper's disaggregated findings, with pinned data
versions and seeds, so a reader can re-run the harm claim.
- Deposit released data or a public archive in a persistent location (e.g. a DOI-issuing repository)
for the camera-ready; keep it consistent with the datasheet.
- For model-generated or scraped inputs, cache raw outputs and record model IDs, dates, and terms —
a study that needs a live API or a since-changed website re-samples rather than reproduces.
Anonymized review version vs. public release
- At submission: any documentation or code shipped for reviewers must be anonymized — no
author names, institution paths, cluster URLs, or identity-revealing repositories, and the
Positionality statement stays out entirely (it is not anonymous).
- After acceptance: replace anonymized placeholders with the public, licensed, persistently
archived versions the camera-ready cites, and finalize the datasheet/model card so it matches the
released artifact exactly.
Consistency with the paper's harm claims
The artifact's job at FAccT is to make the paper's accountability claims checkable. Every
disparity, harm, or transparency benefit the paper asserts should be traceable into the
documentation or released analysis. A model card whose disaggregated numbers disagree with the
paper's table, or an impact assessment that omits the harm a reviewer can foresee, undercuts the
paper more than having no artifact at all.
Vignette: an audit paper's artifact set
A paper auditing a commercial classifier ships: a datasheet for the evaluation dataset (how
assembled, subgroup composition, consent basis); a model card-style report for the audited
system as the authors understand it (intended use, disaggregated error, failure groups); the
audit code with pinned data and seeds regenerating each subgroup table; and a short impact
assessment naming who is harmed by both the system and by publishing the audit, with mitigations.
All anonymized for review, all public and licensed at camera-ready, all consistent with the paper's
tables.
Calibration
- FAccT's artifact expectations are documentation- and release-centered, not badge-centered; do
not import a Docker-image/badge checklist as if it were the bar.
- Whether any optional artifact appendix, reproducibility checklist, or badge exists is
cycle-volatile — confirm on the current Author Guide (待核实).
Output format
[Genres needed] <datasheet / model card / data statement / audit report / impact assessment>
[Artifact role] anonymized review version / public release
[Contents] <provenance / disaggregation / intended-use / limits / license>
[Claim mapping] <paper harm claim -> where in the documentation/analysis it is checkable? yes/no>
[Consistency] <artifact numbers match the paper's tables? yes/no>
[Fixes before upload] <ordered list, kept anonymous for review>
1---2name: facct-artifact-evaluation3description: Use when preparing the accountability artifacts that accompany an ACM FAccT paper — datasheets for datasets, model cards, data statements, audit and impact-assessment documentation, and released code/data — since FAccT's culture centers documentation and accountability infrastructure rather than a formal ACM artifact-badging track; covers what makes each genre credible, anonymized-review versus public-release versions, and consistency with the paper's harm claims.4---56# FAccT Artifact Evaluation78Use this for the material that backs a FAccT paper's transparency and accountability claims. Note9the venue difference up front: FAccT does **not** run the SIGSOFT-style ACM Artifact Review and10Badging track that software-engineering venues use, and it does not hand out Available/Functional/11Reusable/Reproduced badges. **待核实**: confirm on the current Author Guide whether any optional12artifact/reproducibility appendix or badge scheme has been added for your cycle. What FAccT *does*13have is a strong norm of **accountability documentation** — datasheets, model cards, data14statements, audit trails, and impact assessments — plus released code and data. Treat those genres15as your artifact and make each one credible on its own.1617## The FAccT documentation genres (know which your paper needs)1819| Genre | What it documents | When your paper needs it |20|---|---|---|21| Datasheet for a dataset | Motivation, composition, collection, preprocessing, uses, distribution, maintenance | You release or rely on a dataset |22| Model card | Intended use, training data, evaluation **disaggregated by group**, ethical considerations, limits | You release or audit a model |23| Data statement (for language data) | Speaker/annotator demographics, curation rationale, language variety | You build or study a text/NLP corpus |24| Audit / evaluation report | Method, subgroup metrics, thresholds, what was and was not tested | Your contribution is an audit |25| Impact / risk assessment | Foreseeable harms, affected populations, mitigations, residual risk | Deployment or dual-use is plausible |2627Pick the genres your claims actually require; a model audit with no model card, or a dataset paper28with no datasheet, reads as incomplete to this community.2930## What a credible documentation artifact contains3132```text33[Provenance] where the data/model came from, when, under what terms and consent34[Composition] who/what is in it, who is absent, and the resulting blind spots35[Disaggregation] evaluation broken out by protected/affected subgroup, with uncertainty36[Intended use] what it is for — and an explicit "off-label" / do-not-use list37[Limits & harms] known failure groups and foreseeable adverse impacts, not just accuracy38[Maintenance] who updates it, how issues are reported, how long it persists39[License] a clear license for released code/data so others can lawfully reuse it40```4142## Released code and data (the reproducibility half)4344- Ship the analysis that turns data into the paper's disaggregated findings, with pinned data45 versions and seeds, so a reader can re-run the harm claim.46- Deposit released data or a public archive in a persistent location (e.g. a DOI-issuing repository)47 for the camera-ready; keep it consistent with the datasheet.48- For model-generated or scraped inputs, cache raw outputs and record model IDs, dates, and terms —49 a study that needs a live API or a since-changed website re-samples rather than reproduces.5051## Anonymized review version vs. public release5253- **At submission:** any documentation or code shipped for reviewers must be **anonymized** — no54 author names, institution paths, cluster URLs, or identity-revealing repositories, and the55 Positionality statement stays out entirely (it is not anonymous).56- **After acceptance:** replace anonymized placeholders with the public, licensed, persistently57 archived versions the camera-ready cites, and finalize the datasheet/model card so it matches the58 released artifact exactly.5960## Consistency with the paper's harm claims6162The artifact's job at FAccT is to make the paper's **accountability claims checkable**. Every63disparity, harm, or transparency benefit the paper asserts should be traceable into the64documentation or released analysis. A model card whose disaggregated numbers disagree with the65paper's table, or an impact assessment that omits the harm a reviewer can foresee, undercuts the66paper more than having no artifact at all.6768## Vignette: an audit paper's artifact set6970A paper auditing a commercial classifier ships: a **datasheet** for the evaluation dataset (how71assembled, subgroup composition, consent basis); a **model card**-style report for the audited72system as the authors understand it (intended use, disaggregated error, failure groups); the73**audit code** with pinned data and seeds regenerating each subgroup table; and a short **impact74assessment** naming who is harmed by both the system and by publishing the audit, with mitigations.75All anonymized for review, all public and licensed at camera-ready, all consistent with the paper's76tables.7778## Calibration7980- FAccT's artifact expectations are **documentation- and release-centered**, not badge-centered; do81 not import a Docker-image/badge checklist as if it were the bar.82- Whether any optional artifact appendix, reproducibility checklist, or badge exists is83 cycle-volatile — confirm on the current Author Guide (**待核实**).8485## Output format8687```text88[Genres needed] <datasheet / model card / data statement / audit report / impact assessment>89[Artifact role] anonymized review version / public release90[Contents] <provenance / disaggregation / intended-use / limits / license>91[Claim mapping] <paper harm claim -> where in the documentation/analysis it is checkable? yes/no>92[Consistency] <artifact numbers match the paper's tables? yes/no>93[Fixes before upload] <ordered list, kept anonymous for review>94```
brycewang-stanford