thedixitjain WSDM Artifact Evaluation
Package artifacts for a venue where they are persuasion, not process. The pack found no formal artifact-evaluation track or badge system for current WSDM editions (待核实 each cycle) - the CFP-level expectation is the community norm of "practical yet principled": reviewers reward submissions whose claims a skeptic could re-derive. That means the artifact's job is to be inspectable during review and usable after publication, with no committee to certify it.
The reviewer-facing artifact
Because appendices count against WSDM's page budget and there is no rebuttal in which to hand over materials later, the anonymous repository referenced in the PDF is the only expansion space you get. Build it to be skimmed in ten minutes:
anon-artifact/
├── README.md # 1 screen: claim -> script -> expected output table
├── LICENSE # anonymized placeholder license during review
├── env/ # lockfile or container spec, exact versions
├── data/
│ ├── public/ # download scripts for public benchmarks
│ └── PROPRIETARY.md # honest statement of what cannot be shared and why
├── src/ # training/ranking/mining code, no company paths
├── configs/ # one config per reported table row
└── results/
└── seeds_1-5/ # raw metric dumps behind every table in the paper
Anonymity requirements are stricter than habit: fresh remote with no commit
history, no author handles in configs or notebook metadata, no internal package
registries, no dataset paths like /nfs/companyname/.... An Associate Chair
can see who you are; your reviewers must not.
The proprietary-log dilemma
WSDM's core subject matter - query logs, click data, user graphs, ad interactions - is often legally unshareable. The venue's reviewers know this; what they punish is unverifiable work, not industrial work. Choose a rung and state it explicitly in the paper:
| Rung | What is released | What the paper must then carry |
|---|---|---|
| Full release | Data + code + configs | Just the pointers |
| Sampled/anonymized release | A privacy-scrubbed sample + full code | The scrubbing procedure and how sample results track full-data results |
| Public-benchmark mirror | Code + experiments re-run on public datasets | Both result sets, with the deltas discussed, not hidden |
| Code-only | Pipeline code, no data | Enough dataset statistics that a platform-holder could replicate |
| Nothing sharable | - | A candid limitation; expect a proportional credibility discount |
The public-benchmark mirror rung is the WSDM workhorse: pair the proprietary result with the same method on public data so at least one full row of the evidence is end-to-end reproducible by anyone.
Public substitutes worth knowing
- Standard IR/rec benchmarks with public tooling (MS MARCO, TREC collections, MovieLens/Amazon review dumps, KuaiRec-style log releases) - verify current licensing before citing a release plan.
- WSDM Cup datasets: the conference's own annual challenge (the 2027 site already carries a WSDM Cup call) periodically produces real industrial datasets; using one signals community alignment. Check the specific cup's redistribution terms.
- Graph/social benchmarks (SNAP collections and successors) for the network side of the venue.
Anything here can go stale or change license - re-verify at packaging time.
Model and prompt artifacts
Foundation-model-era WSDM papers ("Search with Foundation Models" is in-scope per the 2026 CFP) add artifact surfaces older guidance misses:
- Pin model identity precisely: provider, model string, version/date, and decoding parameters; an unpinned API model makes results unreproducible by construction - say what you did about that (snapshotting outputs, local checkpoints, fixed eval windows).
- Ship prompts verbatim in the repository even when the page budget forces the paper to summarize them.
- Cache and release model outputs used in evaluation when terms permit; that lets others audit the judgment layer without paying for inference.
Credibility signals at a badge-free venue
With no committee stamping artifacts, reviewers use fast proxies to decide whether the repository is real or decorative. Engineer the proxies:
| Signal | Reads as | Cost to provide |
|---|---|---|
| Config file per reported table row | The numbers came from these runs | Minutes |
| Raw seed-level metric dumps | Variance claims are checkable | Minutes |
A make reproduce-table2 entry point |
Someone actually reran this | An hour |
| Download script for each public dataset | The pipeline is end-to-end | An hour |
Honest PROPRIETARY.md |
The authors know what they cannot prove | Minutes |
| Commit dated one hour before deadline as the only commit | Decorative repo | - avoid |
The last row is about the squeeze: a repository assembled on deadline night
tends to mismatch the paper's numbers, and one mismatch found by a reviewer
discounts the entire artifact. Freeze the repo when experiments freeze
(wsdm-workflow sets this at one week out), not when the PDF does.
Post-acceptance conversion
At camera-ready time the anonymous mirror becomes the citable artifact: move to the real organization/account, add the actual license, tag the release that matches the camera-ready numbers, and archive a snapshot (e.g., a DOI-issuing archive) so the URL in the ACM DL version outlives your CI. Update the README's claim-to-script table against final camera-ready table numbers - drift between repo and proceedings is the most common post-publication complaint.
One more conversion detail: if the paper used the public-benchmark-mirror rung, keep both pipelines in the public repo permanently - the mirror is what future papers will actually build on, and its issues page becomes your citation engine.
Output format
[Artifact tier] full / sampled / public-mirror / code-only / none + stated in paper?
[Ten-minute test] README claim->script->output table present: yes / no
[Anonymity sweep] history / handles / paths / registries: clean or leaks listed
[FM pins] model string+date+decoding params recorded: yes / no / n-a
[Post-acceptance plan] public home, license, tagged release, archival snapshot
Source: brycewang-stanford/Awesome-Journal-Skills → WSDM-Skills/skills/wsdm-artifact-evaluation/SKILL.md