thedixitjain WSDM Reproducibility
Make a web-search/data-mining paper re-derivable. WSDM has no reproducibility checklist to fill (none surfaced for current editions; 待核实 each cycle) - which raises the bar rather than lowering it, because reviewers apply the norm without giving you a form to hide behind. The venue-specific twist: WSDM evidence usually comes from behavioral data (queries, clicks, follows, purchases), and behavioral data has failure modes that generic ML reproducibility advice never mentions.
Provenance: behavioral data decays
A log dataset is a measurement of a platform at a moment - the platform's ranker, UI, and user base are all baked into it. Reproducibility therefore starts with recording what generated the data:
- Collection window, platform surface (web vs app, market/locale), and any known ranker or UI changes inside the window.
- The logging policy: what produced the exposures users could click on. A click log is a logged-policy artifact; results on it are conditional on that policy (this is the entire lesson of the position-bias and unbiased learning-to-rank literature born at this venue).
- Filtering steps with counts at every stage: bots removed, sessions segmented, minimum-activity thresholds. Two labs "using the same dataset" routinely diverge purely on preprocessing counts.
Temporal discipline
Random splits on interaction data leak the future into training. Default to time-based splits and document them to the day:
# split-manifest.yaml - ship with the artifact, cite in the paper
dataset: platform-logs-v3
train: {start: 2025-01-06, end: 2025-05-31}
valid: {start: 2025-06-01, end: 2025-06-14}
test: {start: 2025-06-15, end: 2025-06-28}
user_handling: users may span splits (temporal, not user-disjoint)
item_handling: cold items in test retained; reported separately
leakage_checks:
- no feature computed over any window overlapping valid/test
- global statistics (IDF, popularity) frozen at train end
notes: one ranker deployment change on 2025-04-12 inside train window
State whether users are shared across splits (temporal split) or disjoint (generalization-to-new-users split) - the two answer different questions and mixing them is a classic silent irreproducibility source in recommendation papers.
Bias assumptions are part of the method
If the paper estimates relevance or preference from clicks, its results depend on an exposure/position-bias model. Reproducibility means naming it:
| What you assume | What must be reported |
|---|---|
| Position bias (examination model) | The propensity model, how it was estimated, on what data |
| No exposure bias (rare, say so) | Why the setting justifies it |
| Popularity/selection bias corrected | The correction estimator and its hyperparameters |
| Offline metrics proxy online value | The known gap, plus any online evidence |
An unstated bias model makes the numbers unreproducible even with the code, because a re-implementer will pick a different default.
Runs, seeds, and variance
- Report the number of runs and the seed policy for every learned component; ranking metrics on sparse test sets are noisy, and single-run nDCG deltas of under a point are routinely within seed variance.
- Give variance (std or CI) for headline comparisons; where a system-scale experiment genuinely cannot be repeated, say "single run" in the table note rather than letting the reader assume otherwise.
- Statistical tests over query/user-level paired differences beat aggregate deltas; state the unit of analysis (query, session, user) - it changes the test.
The honesty ladder for unrerunnable results
Industrial WSDM papers often include numbers nobody outside can regenerate (online A/B tests, full-traffic logs). Use graded language that matches the evidence tier, and put the tier in the paper:
- Rerunnable: public data + released code; anyone can regenerate tables.
- Rebuildable: released code + documented proprietary pipeline; an insider could regenerate, an outsider can audit the logic.
- Attested: online/production results reported with measurement protocol (traffic share, duration, metric definitions, guardrails) but not regenerable. Attested numbers support deployment claims, not method-ranking claims - do not let an A/B win stand in for a missing offline comparison.
Privacy is a reproducibility constraint, not an excuse
WSDM requires an ethical-considerations section; user-data handling belongs in
it. De-identification, aggregation thresholds, and consent/ToS basis for the
data should be stated - and any released sample must survive a re-identification
sniff test (rare queries and long-tail items are quasi-identifiers). "We cannot
release anything" is acceptable only alongside rung 2-3 evidence above and a
public-benchmark mirror where feasible (see wsdm-artifact-evaluation).
Pre-submission reproducibility sweep
Run once when experiments freeze, once on the final PDF:
[ ] Data provenance paragraph: window, surface, logging policy, filters+counts
[ ] Split manifest shipped and cited; user-sharing across splits stated
[ ] Bias/exposure model named, with estimation procedure and data
[ ] Seeds and run counts per learned component; variance on headline deltas
[ ] Unit of analysis named for every statistical test
[ ] Each result family labeled: rerunnable / rebuildable / attested
[ ] Attested results carry protocol: traffic %, duration, metric definitions
[ ] Released sample re-identification check done (rare queries, tail items)
[ ] Ethics section covers user-data basis and mitigations, specifically
[ ] Repo numbers regenerate paper tables (spot-check two tables end-to-end)
Items that fail with no time to fix become limitation sentences, not silence - at a no-rebuttal venue, a disclosed gap is survivable and a discovered one usually is not.
Output format
[Provenance] window / surface / logging policy / filter counts: recorded?
[Splits] temporal manifest present; user-sharing stated: yes / no
[Bias model] named + estimation reported: yes / no / not applicable
[Variance] runs, seeds, CI/test + unit of analysis per headline table
[Tier] rerunnable / rebuildable / attested per result family
[Privacy] ethics-section coverage of user data: adequate / gaps listed
Source: brycewang-stanford/Awesome-Journal-Skills → WSDM-Skills/skills/wsdm-reproducibility/SKILL.md