RecSys Artifact Evaluation
Use this for evidence packaging around RecSys. The venue does not run a separate artifact-badge
process; instead the Call for Contributions expects a link to an anonymous repository inside
the paper, and reproducibility-minded reviewers judge the paper partly by whether that repository
makes the ranking claims regenerable.
Artifact plan
- Decide what a reviewer needs to trust the claim: the exact dataset version, the split script,
baseline configurations, tuning grids, trained-model checkpoints, exposure/propensity logs for
off-policy claims, seeds, and the evaluation code.
- Keep decision-critical evidence in the paper or appendix; optional run files live in the
repository, because RecSys reviewers are not obliged to open it.
- Anonymize repository history, commit authors, cluster paths, license headers, and any platform
or organization names.
- Include a one-minute reproduction map: environment, dependencies, dataset download or
identifier, commands, expected ranking numbers, runtime, seeds, and known nondeterminism.
- For proprietary interaction data, give enough provenance and preprocessing detail for credible
reproduction on a public dataset without violating data-use terms.
- After acceptance, swap the anonymous mirror for a public, licensed, citable archive.
What RecSys evidence reviewers open first
| Claim type |
First artifact inspected |
Common failure caught |
| Top-N ranking gain |
Split script + baseline configs |
Random split leaking the future; baselines under-tuned |
| Off-policy / counterfactual result |
Logged propensities + estimator code |
Propensities missing, so the IPS/DR estimate cannot be recomputed |
| Sequential/session model |
Data ordering + leave-one-last split |
Test interactions seen during training |
| Reported metric values |
The scorer and its cutoff |
Sampled metrics presented as full-ranking numbers |
Because RecSys reviewers can and do re-run a small offline pipeline, make the headline ranking
table regenerable with one command before polishing anything else.
Worked vignette: packaging an off-policy study
A hypothetical submission proposes an exposure-corrected ranker validated offline and in a
semi-synthetic simulator.
- Ship the logging policy's propensities alongside the interactions, not just the clicks, so
the inverse-propensity estimate can be recomputed.
- Provide the simulator as one parameterized script so a reviewer can vary exposure strength and
reward, rather than trusting a single frozen curve.
- Emit every ranking and reward table directly from logged results so the PDF numbers and the
repository numbers cannot drift apart.
- State which regime the simulator satisfies the positivity assumption in and where it breaks it,
since that mapping is what reproducibility-minded reviewers grade.
Calibration anchors
repo/
README.md # one-minute orientation: env, data id, one command per table
environment.yml # pinned versions of the recommender framework and deps
data/PREP.md # dataset version, split protocol (temporal), checksums
configs/ # per-model + per-baseline configs with tuning grids
run_tables.sh # regenerates Table 1..N from seeds
- Assume only the README and one entry script get opened; design for that.
- Repository size limits and accepted hosting change by cycle; verify against the current
submission instructions rather than a past year.
Output format
[Artifact role] anonymous in-paper repo / camera-ready public archive
[Contents] <data/splits/configs/checkpoints/propensities/seeds>
[Anonymity risks] <paths / commit authors / platform names / URLs>
[Reproduction level] turnkey / scripted / descriptive / weak
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → RecSys-Skills/skills/recsys-artifact-evaluation/SKILL.md
1---2name: recsys-artifact-evaluation3description: Use when packaging ACM RecSys code, datasets, splits, trained models, propensity logs, and seeds as an anonymous in-paper repository during review or a public archive after acceptance, even though RecSys has no separate artifact badge — covering what recommender reviewers open first and how to make a top-N ranking table regenerable end to end.4---567# RecSys Artifact Evaluation89Use this for evidence packaging around RecSys. The venue does not run a separate artifact-badge10process; instead the Call for Contributions expects a link to an **anonymous repository** inside11the paper, and reproducibility-minded reviewers judge the paper partly by whether that repository12makes the ranking claims regenerable.1314## Artifact plan1516- Decide what a reviewer needs to trust the claim: the exact dataset version, the split script,17 baseline configurations, tuning grids, trained-model checkpoints, exposure/propensity logs for18 off-policy claims, seeds, and the evaluation code.19- Keep decision-critical evidence in the paper or appendix; optional run files live in the20 repository, because RecSys reviewers are not obliged to open it.21- Anonymize repository history, commit authors, cluster paths, license headers, and any platform22 or organization names.23- Include a one-minute reproduction map: environment, dependencies, dataset download or24 identifier, commands, expected ranking numbers, runtime, seeds, and known nondeterminism.25- For proprietary interaction data, give enough provenance and preprocessing detail for credible26 reproduction on a public dataset without violating data-use terms.27- After acceptance, swap the anonymous mirror for a public, licensed, citable archive.2829## What RecSys evidence reviewers open first3031| Claim type | First artifact inspected | Common failure caught |32|---|---|---|33| Top-N ranking gain | Split script + baseline configs | Random split leaking the future; baselines under-tuned |34| Off-policy / counterfactual result | Logged propensities + estimator code | Propensities missing, so the IPS/DR estimate cannot be recomputed |35| Sequential/session model | Data ordering + leave-one-last split | Test interactions seen during training |36| Reported metric values | The scorer and its cutoff | Sampled metrics presented as full-ranking numbers |3738Because RecSys reviewers can and do re-run a small offline pipeline, make the headline ranking39table regenerable with one command before polishing anything else.4041## Worked vignette: packaging an off-policy study4243A hypothetical submission proposes an exposure-corrected ranker validated offline and in a44semi-synthetic simulator.4546- Ship the logging policy's **propensities** alongside the interactions, not just the clicks, so47 the inverse-propensity estimate can be recomputed.48- Provide the simulator as one parameterized script so a reviewer can vary exposure strength and49 reward, rather than trusting a single frozen curve.50- Emit every ranking and reward table directly from logged results so the PDF numbers and the51 repository numbers cannot drift apart.52- State which regime the simulator satisfies the positivity assumption in and where it breaks it,53 since that mapping is what reproducibility-minded reviewers grade.5455## Calibration anchors5657```text58repo/59 README.md # one-minute orientation: env, data id, one command per table60 environment.yml # pinned versions of the recommender framework and deps61 data/PREP.md # dataset version, split protocol (temporal), checksums62 configs/ # per-model + per-baseline configs with tuning grids63 run_tables.sh # regenerates Table 1..N from seeds64```6566- Assume only the README and one entry script get opened; design for that.67- Repository size limits and accepted hosting change by cycle; verify against the current68 submission instructions rather than a past year.6970## Output format7172```text73[Artifact role] anonymous in-paper repo / camera-ready public archive74[Contents] <data/splits/configs/checkpoints/propensities/seeds>75[Anonymity risks] <paths / commit authors / platform names / URLs>76[Reproduction level] turnkey / scripted / descriptive / weak77[Fixes before upload] <ordered list>78```7980---8182**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `RecSys-Skills/skills/recsys-artifact-evaluation/SKILL.md`
thedixitjain