brycewang-stanford NAACL Artifact Evaluation
NAACL has no separate artifact-badging track; artifacts are judged inside the ARR review itself, through the supplement upload and section B of the Responsible NLP checklist ("scientific artifacts"). That placement matters: your artifact documentation is not an optional extra but a set of sworn answers reviewers cross-examine against the PDF.
What the artifact must let a reviewer do
- Trace provenance. Where did every corpus come from, under what license, and does your use match the terms and the creators' intent?
- Inspect the instrument. Prompts, annotation guidelines, interface screenshots, and pay rates are artifacts too — a human-evaluation claim without its instrument is unverifiable.
- Rerun the cheap parts. Scoring scripts and metric code should execute from the archive alone; nobody will retrain your model, but everybody can re-score your outputs if you include them.
- Audit the data card. Language varieties, dialect coverage, speaker demographics where relevant, known gaps, and intended use.
Language-data documentation ladder
| Data situation | Minimum documentation for a NAACL reviewer | Extra step |
|---|---|---|
| Standard public benchmark | Version, split, license, citation | Note any known contamination reports |
| Web-scraped text | Collection dates, filtering rules, deduplication, license basis | PII handling statement |
| New annotated corpus | Guidelines, annotator recruitment and pay, agreement scores | Release the guidelines verbatim in the supplement |
| Dialectal / code-switched data | Variety labels and how they were assigned | Native-speaker validation description |
| Indigenous or community-owned language data | Consent and partnership terms, community approval for release | Verify whether public release is permitted at all |
The last row is a NAACL signature concern. Work on languages of the Americas increasingly follows community-controlled data norms: some corpora may be used but not redistributed, some require named attribution (which conflicts with anonymous review — use a placeholder and restore at camera-ready), and some communities set conditions on derived models. "We release everything" is not automatically the ethical high ground here; the checklist rewards accuracy about constraints, not maximal openness.
Anonymous packaging that survives inspection
artifact.zip
├── README.md # one-screen orientation: what, how, how long
├── data/
│ ├── data_card.md # provenance, license, varieties, gaps
│ └── samples/ # enough rows to judge quality, not the corpus
├── prompts/ # exact strings, all variants tried
├── eval/
│ ├── score.py # runs on outputs/ with no network access
│ └── outputs/ # raw model outputs backing the main tables
└── annotation/
└── guidelines.pdf # the instrument, scrubbed of institution marks
Scrub before zipping: repository history, notebook execution metadata, absolute paths with usernames, license headers naming the lab, and any consent form carrying institutional letterhead (replace with a redacted copy; note that the original exists).
Vignette: a Quechua-Spanish parallel corpus package
A submission introduces a 40k-pair Quechua-Spanish parallel corpus built with two community organizations, plus MT baselines. The packaging calls that follow from this skill:
- The corpus itself does not ship in the review archive — the partnership terms permit research use but defer public release to a community decision. The data card states this, and section B of the checklist answers "no, with reason" for artifact release.
- What does ship: 200 sample pairs cleared for review purposes, the collection protocol, annotator recruitment and payment description, the cleaning scripts, and the full MT evaluation pipeline with outputs.
- The consent-form template ships with the letterhead redacted and a note that the original is held by the partner organizations.
- The README's first paragraph tells the reviewer exactly which claims the archive can and cannot let them verify — pre-empting the "authors refuse to release data" misreading with a governance explanation instead.
The result is an artifact that scores as honest and inspectable rather than incomplete, which is the realistic best outcome for community-governed data.
Cycle-volatile mechanics
Archive size caps, accepted formats, and whether supplements upload as one file or several are OpenReview-form details that shift between ARR cycles; read the live submission form before building the final zip, and never reverse-engineer the limits from a previous cycle's folklore.
Post-acceptance conversion
At camera-ready, the anonymous bundle becomes the public record: move it to a persistent host with a DOI or a tagged release, apply the real license, restore attribution the community partnership requires, and update the checklist-facing statements in the paper if the release scope changed between review and publication.
Output format
[Artifact inventory] <data / code / prompts / guidelines / outputs>
[Checklist B alignment] <each B answer -> where the artifact proves it>
[Provenance gaps] <unlicensed, undocumented, or unclear-consent items>
[Community constraints] <redistribution / attribution / approval terms>
[Anonymity sweep] clean / issues found
[Release plan] <anonymous now -> public form at camera-ready>