brycewang-stanford Web Conference Artifact Evaluation
The Web Conference's 2026 artifact program was an availability badge, not a reproducibility audit: a volunteer committee performed a light verification that the artifact exists, is downloadable from a publicly accessible archival repository, and carries minimal instructions. Accepted papers from all tracks of the main or companion proceedings — except workshop papers — could apply, with the artifact submitted at camera-ready time. Design the artifact to clear that bar first, then exceed it for your actual readers.
What the badge checks vs. what readers need
| Dimension | Badge bar (2026) | Reader bar |
|---|---|---|
| Location | Public archival repository | Same, plus a mirror for big files |
| Persistence | Link resolves at check time | DOI/versioned, survives repo moves |
| Instructions | "Minimal" access instructions | Environment, run commands, expected output |
| Completeness | Artifact "exists" | Enough to regenerate headline tables |
| Legality | Not assessed | Licenses and platform ToS actually permit sharing |
"Archival" is the operative word: a personal GitHub repository can be deleted or rewritten and historically has not counted as archival for ACM badging purposes. Deposit a release snapshot in Zenodo, figshare, or an institutional repository that mints DOIs, and let GitHub be the development mirror the README points to.
Web-data artifacts are legally different
This venue's artifacts are dominated by crawls, platform datasets, interaction logs, and derived embeddings — objects with third-party rights attached. Decide the sharing tier per component before promising anything in the paper:
- Freely shareable: your code, your synthetic data, aggregate statistics.
- Shareable as derivatives: URL lists, item IDs, hydration scripts — release the pointers plus the pipeline, not the content, when platform terms forbid redistribution (the standard pattern for social-media datasets).
- Shareable on request / gated: user-level data behind a data-use agreement; document the request path in the README so the artifact is still "available."
- Not shareable: proprietary logs. Say so in the paper's availability statement and ship the measurement code anyway.
A takedown-resilient design states the crawl window, preserves checksums of the original corpus, and includes the re-crawl script — so when 15% of URLs die (they will), a later user can quantify exactly what changed.
Minimum viable package
artifact-v1.0/ (deposited at Zenodo, DOI 10.5281/zenodo.XXXXXXX)
├── README.md # what this is, paper title, 3-command quickstart
├── LICENSE # code license + per-dataset terms table
├── environment.yml # or Dockerfile / requirements.txt with pins
├── data/
│ ├── MANIFEST.md # per-file: source, crawl window, row counts, sha256
│ └── ids/ # hydration pointers where content can't ship
├── src/ # training/analysis code, seeds surfaced as flags
├── scripts/
│ ├── reproduce_table2.sh
│ └── recrawl.py # dead-link accounting for future users
└── CITATION.cff
The single highest-leverage file is reproduce_table2.sh — one command per
headline result. Badge reviewers may not run it; the citing researcher eighteen
months later definitely will.
During review vs. after acceptance
Review at this venue is double-blind and the appendix is the sanctioned home for
reproducibility detail, so during review the artifact appears as an anonymized
link (if at all) with no author-identifying paths, git history, or platform account
names. After acceptance, the flow inverts: mint the DOI first, cite it in the
camera-ready, then file the badge application alongside the final files — the
badge's availability check needs the link that will live in the published paper.
Sequence details sit in webconf-camera-ready; content standards for what the
package proves sit in webconf-reproducibility.
Vignette: the crawl that got a takedown notice
A team releases a 40M-page news crawl with their WWW paper. Eight months later, a publisher demands removal of its articles. Because the artifact followed the tiered design, the response costs an afternoon, not the artifact: the Zenodo deposit v1.1 removes the publisher's content files, the manifest documents the removal (URLs retained as pointers, checksums preserved), the hydration script still lets rights-holding users rebuild the full corpus, and the paper's DOI citation now resolves to a version history that explains itself. The counterfactual design — one monolithic tarball with raw content — would have forced a full withdrawal and orphaned every downstream citation. Design for the takedown on day one; on the Web it is a when, not an if.
Badge application dry run
Before filing with the camera-ready, simulate the committee's light check from a clean machine and a logged-out browser:
- The link in the paper resolves to the archival record (not a login page, not a personal homepage redirect).
- The record shows a version, a license, and a README visible without download.
- One file downloads successfully and matches its manifest checksum.
- The README's first screen states what the artifact is and which paper it accompanies — the committee volunteer decides in minutes.
Honest labeling
The Artifacts Available badge certifies existence, not correctness. Do not write "our results are independently verified" on the strength of it. Conversely, an artifact too encumbered to badge (tier 3-4 data) does not bar publication — the paper just needs an availability statement that says precisely what is released, what is gated and why, and what a reader can still verify with the released parts.
Output format
[Artifact tiering] code=<tier> data=<tiers per set> models=<tier>
[Badge eligibility] track eligible? archival deposit? DOI minted?
[Legal check] platform ToS / license conflicts found: <list or none>
[Decay plan] crawl window stated? checksums? recrawl script?
[One-command repro] present for headline tables: yes/no
[Availability statement] drafted for the paper: yes/no