Sigir Reproducibility

SIGIR Reproducibility

Reproducibility is unusually load-bearing at SIGIR for two reasons. First, the venue runs a dedicated Reproducibility track (its own track in 2026, split out of the former combined Resource & Reproducibility track — budget and dates 待核实 on the current page), so reproduction studies are publishable first-class work. Second, the field's own literature documents how often reported IR gains fail to replicate under matched tuning — reviewers of regular papers therefore read reproducibility signals as a proxy for whether the gains are real.

Why IR results drift: the usual suspects

A reproducibility-strong SIGIR paper closes each row with an artifact, not a promise: the config file is the documentation.

Minimum reporting block for any empirical SIGIR paper

Put this in the paper (it fits in ~0.3 page and pre-empts three review objections):

• Collections with versions; train/dev/test usage; any filtering.
• Index/build settings for every system, baselines included.
• Tuning protocol: search space, budget, selection metric, dev split — symmetric across systems.
• Seeds: how many, and whether tables show mean, sd, and the significance test.
• Compute: hardware, wall-clock for index+train+retrieve, and query latency setup.
• Pointer to the repository with run files (see sigir-artifact-evaluation).

# config-as-artifact: one file per reported system, committed to the repo
system: ours-dense-v2
collection: msmarco-passage/dev/small        # ir_datasets id
index: {tokenizer: bert-base-uncased, max_len: 256, stride: 128}
train: {seeds: [13, 42, 71], batch: 64, lr: 2e-5, epochs: 3}
eval: {tool: ir_measures, metrics: [nDCG@10, RR@10, R@1000], qrels: official}
significance: {test: paired-t, correction: bonferroni, alpha: 0.05}

Writing a Reproducibility track paper

A reproduction study is not a re-run; it is an investigation. The track rewards:

1. Target selection with stakes — results the community builds on (a widely cited ranker, a standard baseline configuration, a claimed efficiency win).
2. Faithful reimplementation first: reproduce with the original artifacts where they exist; document every forced deviation and why.
3. Divergence analysis as the contribution: when numbers differ, isolate the cause (collection version? tuning? eval tool?) with controlled toggles — the drift table above is your experimental design.
4. Generalization probes: does the original conclusion survive new collections, matched tuning, or current baselines? "Holds, but the margin halves under equal tuning" is a publishable, community-serving finding.
5. Respectful register: the genre convention is scientific, not gotcha — criticize configurations, not authors, and give original authors' artifacts credit where due.

Anti-patterns the track's reviewers flag: reproducing only the headline number while skipping ablations; declaring "failure to reproduce" without exhausting configuration space; and shipping a reproduction whose own pipeline is unpinned (the irony reject).

Decay planning: the three-year audience

A SIGIR paper's reproducibility has two audiences with different failure modes: the reviewer this spring (needs the 10-minute runnable path) and the researcher in three years (meets link rot, dataset takedowns, deprecated toolkit APIs, and vanished model checkpoints). Plan for the second audience explicitly:

• Cite collections by stable identifiers (ir_datasets ids, TREC track names, dataset DOIs), never by lab-server URLs.
• Freeze an archival copy of your own artifacts (Zenodo/institutional DOI) at camera-ready; GitHub is a working mirror, not an archive.
• Record toolkit versions in the paper text, not only in the lockfile — the paper outlives the repository more often than authors expect.
• If a resource you depend on has restrictive terms (query logs, commercial APIs), state what a future reproducer can do without it: which tables survive, which die.
• Leave the qrels/run checksums in the repo README; three years later they are the only way to prove a re-download matches the evaluated state.

Reproducibility as review defense for regular papers

• Close comparisons without variance are the most common SIGIR experimental criticism; three seeds with mean ± sd is cheap insurance for neural systems.
• An honest "we could not reproduce baseline X's published number; we report our best faithful configuration (details in repo)" reads as strength, not weakness — the community knows the drift problem intimately.
• Never copy baseline numbers across collections or eval setups from other papers' tables without saying so; mixed-provenance tables are a known reject trigger.

Quick self-audit before submission

• Every number in every table regenerates from a shipped run file plus one documented command.
• Every system row has a config file; diffs between systems are visible as config diffs, not prose.
• The seeds behind each neural row are enumerable, and the shipped run is identified (which seed, or the per-topic mean).
• A colleague outside the project reproduced Table 1 from the repo README without asking questions (the strongest cheap test available).
• The paper's reporting block and the repo's configs agree — reviewers diff them when suspicious.

Output format

[Mode] hardening a regular paper / Reproducibility track study
[Drift audit] rows closed with artifacts: <k>/7 (collection/index/processing/pool/seed/tool/tuning)
[Reporting block] present in paper y/n; missing items <list>
[For repro studies] target + stakes / faithfulness log / divergence causes isolated
[Variance] seeds <n>, mean±sd shown y/n, test named y/n
[Biggest residual risk] <the one unpinned thing a reviewer will find>

Source: brycewang-stanford/Awesome-Journal-Skills → SIGIR-Skills/skills/sigir-reproducibility/SKILL.md