brycewang-stanford RecSys Reproducibility
Reproducibility is unusually load-bearing at RecSys for two reasons. First, the venue runs a dedicated Reproducibility Track for papers that repeat, refute, or re-scope prior results and for datasets and frameworks that enable future reproduction. Second, the field's own literature documented how often reported recommender gains fail to survive a fair re-evaluation, so reviewers of regular papers read reproducibility signals as a proxy for whether the gains are real.
Why recommender results drift: the usual suspects
| Drift source | Typical symptom | Pin it by |
|---|---|---|
| Dataset version / filtering | "MovieLens" numbers differ across papers | Exact version id, k-core filter, and checksums |
| Split protocol | Session results inflated | Temporal or leave-one-last split, not random; document it |
| Baseline tuning asymmetry | Neural method "beats" everything | Equal search budget per system, grid and selected config logged |
| Sampled vs full-ranking metrics | Recall/nDCG off by a lot | Rank over the full item set, or state sampling and cutoff |
| Seeds and nondeterminism | ±0.005 nDCG run to run | Multiple seeds; report mean ± sd, not the best run |
| Metric implementation | nDCG differs at the 3rd decimal | One canonical scorer with cutoff and tie-handling recorded |
| Off-policy propensities | Counterfactual estimate irreproducible | Release logged propensities and the estimator variant (IPS/SNIPS/DR) |
A reproducibility-strong RecSys paper closes each row with an artifact, not a promise: the config file is the documentation.
Minimum reporting block for any empirical RecSys paper
Put this in the paper (it fits in ~0.3 page and pre-empts three review objections):
- Datasets with versions and filtering; train/validation/test usage; the split protocol.
- Tuning protocol: search space, budget, selection metric, validation split — symmetric across systems, baselines included.
- Metrics: which ones, the cutoff, and whether ranking is over the full item catalog or a sampled candidate set.
- Seeds: how many, and whether tables show mean, sd, and a significance test on close results.
- For any deployment or off-policy claim: the estimator, its assumption, and the logged propensities.
The Reproducibility Track specifically
A reproduction paper is first-class work here, not a lesser contribution.
[reproduction scope] which prior papers, which claims, which datasets
[matched setup] identical splits, metrics, and — critically — equal tuning budget
[divergence report] where results reproduce, where they do not, and the diagnosed cause
[new context] a new domain, dataset, or baseline that tests generality
[artifact] a runnable pipeline others can extend
Honest divergence is the contribution: "we could not reproduce claim X under matched tuning, and here is why" is publishable when the diagnosis is rigorous.
Vignette: a session-model reproduction
Consider reproducing three published session recommenders. Its reproducibility spine: the exact dataset versions and k-core filter, a temporal split shared across all systems, an equal tuning budget so no method is advantaged by search, full-ranking metrics at a fixed cutoff, five seeds with mean ± sd, and one honest paragraph on the one result that did not reproduce and the preprocessing difference that explains it.
Degrees of reproducibility
- Turnkey: one command regenerates each ranking table from logged seeds.
- Scripted: scripts exist but need documented manual steps or an external dataset download.
- Descriptive: prose detailed enough that a competent reader could rebuild the pipeline.
For RecSys, the offline pipeline should be turnkey because reviewers actually re-run it; a production A/B result may stay descriptive with the deviation documented. State the achieved level honestly rather than overpromising a turnkey build that fails on a clean machine.
Output format
[Claim inventory] <claim -> evidence location>
[Reproducibility status] strong / partial / weak
[Drift risks] <dataset version / split / baseline tuning / sampled metrics / seeds / propensities>
[Paper fixes] <must appear in main PDF>
[Repository fixes] <anonymous-repo additions>