Cikm Reproducibility

CIKM Reproducibility

Reproducibility at CIKM has a venue-specific difficulty: the typical paper chains components from different communities — an index, a graph, a model, a ruleset — and each link has its own silent-divergence habits. A reader who cannot rebuild the chain cannot attribute the result, and a blended review panel contains someone able to notice each weak link.

Divergence map for chained pipelines

The discipline: for each link, either the artifact pins it or the paper states it. A link pinned nowhere is where a failed replication will land.

Unreleasable data, releasable knowledge

CIKM's KM lane routinely involves enterprise corpora, clickstreams, or proprietary KGs that cannot ship. The venue-honest pattern:

• Describe the unreleasable data statistically (size, schema, class balance, collection window) at a level where a reader could construct a synthetic analog — then actually provide that analog generator when feasible.
• Run the public-data variant of every headline experiment, even if the effect is smaller; a result that exists only on invisible data asks the panel for faith.
• State the release position explicitly in the paper ("logs cannot be released; the sampling script and schema are in the artifact") rather than leaving the reader to discover the gap.

GenAI disclosure as a reproducibility document

CIKM 2026's mandatory GenAI Usage Disclosure covers code and data, not just prose (source map, 2026-07-08). Treat it as part of the methods record: if evaluation scripts, synthetic data, prompts, or labels were generated with AI assistance, the disclosure plus the artifact should together let a reader judge what that implies for the result. A disclosure that says "AI used for coding" while the artifact contains unexplained generated labels is an inconsistency automated compliance checks — which the conference reserves — or reviewers can catch.

Environment capture

Chained pipelines multiply environment surface, so capture it in layers:

The test is transferability: a lab-mate on a clean machine, without the authors in the room, reaches the headline table. Running that internal replication before submission is the single highest-yield reproducibility exercise — it finds the unpinned link while it can still be pinned.

Where reproducibility pays at this venue

Three concrete CIKM payoffs beyond principle. First, the blended panel: whichever lane doubts the result will probe its own link of the chain, so pinning every link is defense in all three directions. Second, resource-track reviewers and readers judge adoptability, which is reproducibility wearing its public face (cikm-artifact-evaluation). Third, follow-up work: CIKM's back catalog shows methods becoming standard baselines (DRMM, BERT4Rec); papers get that afterlife only when third parties can run them — the reproducible version of a method is the one that accumulates citations as a baseline.

Release timeline

Anonymized review artifact during submission (see cikm-supplementary for what the budget permits); public repository at camera-ready, with license, versioned release tag, and the exact commit that produced the proceedings numbers. The 2026 notification-to-camera-ready window is thirteen days — build the release during the review wait (cikm-workflow Mode A), not inside that window.

Honest-failure disclosure

Chained pipelines rarely reproduce perfectly, and the venue-credible move is to say so first: a REPRODUCING.md that states which numbers regenerate exactly, which vary within a stated tolerance (GPU nondeterminism, sampling), and which depend on restricted inputs and therefore only regenerate in public-analog form. Declared tolerance reads as competence; discovered variance reads as concealment. The same document is where to state known environment sensitivities ("results verified on CUDA X; version Y shifts Table 3 by ±0.2") — the sentence that saves a replicator a week is the sentence that earns the citation.

One-command bar

# The replication target for a CIKM chained pipeline:
git clone <repo> && cd <repo>
make setup          # pinned environment, data download or synthetic analog
make table2         # rebuilds the headline table end-to-end from the frozen inputs

If make table2 cannot exist because data is restricted, the repo must say so at the top and offer the public-variant target instead. Silent partiality — a repo that looks complete but is not runnable — costs more reviewer goodwill than an honest scope statement.

Output format

[Chain audit] <link → pinned where (paper / artifact / nowhere)>
[Data position] <releasable / described+analog / public-variant-only>
[Disclosure consistency] <GenAI section vs. artifact contents>
[Release plan] <review artifact state → camera-ready repo state, dated>
[Weakest link] <the divergence a replicator would hit first>

Source: brycewang-stanford/Awesome-Journal-Skills → CIKM-Skills/skills/cikm-reproducibility/SKILL.md