Icdm Reproducibility

ICDM Reproducibility

Make the mining result reproducible as science, under two ICDM-specific pressures: the Research Track is triple-blind, so the reproduction package cannot reveal identity, and the reporting all competes for space inside the 10-page all-inclusive cap. Reproducibility at ICDM is not a separate checklist form (verify per edition); it is evidence that the discovery is real rather than a lucky configuration.

What must be reproducible

• The mining task setup: exact datasets, versions, preprocessing, splits, and how ground truth or injections were generated.
• The method: hyperparameters, model selection procedure, the mechanism's knobs (e.g. sketch size, partition count), and the random seeds.
• The evaluation: metrics, thresholds/cutoffs, variance estimation, and the hardware behind any timing claim.

Reproducibility tiers

Aim for rerunnable on any public-data result; use attested only where data genuinely cannot be released, and say so plainly.

Configs as artifacts

Put the run behind a config, not scattered command-line flags, so a reviewer reproduces a table by pointing at a file.

# repro/config.yaml  (anonymized; no author paths, no institutional dataset names)
task: stream_anomaly_ranking
dataset: public_edge_stream_v3      # public source + version, not an internal name
seeds: [0, 1, 2, 3, 4, ... , 19]    # the 20 seeds behind the reported variance
method:
  sketch_partitions: 128            # the mechanism knob mapped in the ablation
  hash_family: multiplicative
eval:
  metric: precision_at_k
  k: 100
  time_respecting_split: true
compute:
  hardware: "1x consumer GPU, 16GB"  # generic; states the basis of any timing claim

Triple-blind the package

• Export a fresh repository with no git history — commit metadata routinely leaks author names and institutions.
• Remove author paths, usernames, internal dataset names, cluster hostnames, and README acknowledgements.
• Host it so the link in the PDF resolves to an anonymized location, not a named account.
• Because ICDM traditionally offers no rebuttal, this package is often the only extra evidence a reviewer ever sees — it must be complete and anonymous at submission time.

Report inside the page cap

• The full reproduction protocol can live in an in-cap appendix and the cited repository; the body must still state seeds, key hyperparameters, and the compute basis of timing claims.
• Prefer a compact reproducibility paragraph plus a repository over a sprawling appendix that eats pages the body needs.

Vignette: the seed table that answered a review before it was written

A team worried reviewers would read their close margins as noise. Rather than hope for a rebuttal that ICDM might not offer, they reported every metric with a standard deviation over 20 seeds, shipped the seed list and configs in an anonymized, history-scrubbed repository cited in the PDF, and stated the exact GPU behind their latency numbers. The "is this just noise?" review never came, because the paper had already answered it — and nothing in the package revealed who they were.

Output format

[Repro tier] rerunnable / rebuildable / attested (per result)
[Seeds+variance] reported: yes / no
[Config-as-artifact] present: yes / no
[Anonymized package] history-scrubbed + no institutional names: yes / leaks found
[Compute basis] hardware behind timing claims stated: yes / no
[Top gap] <single most important missing reproducibility detail>

Source: brycewang-stanford/Awesome-Journal-Skills → ICDM-Skills/skills/icdm-reproducibility/SKILL.md