ICDM Experiments
Design the evaluation an ICDM reviewer will trust: a defined mining task, baselines tuned as
carefully as your method, ablations that isolate the mechanism, a measured scale story, and a
discovery-validity argument. ICDM's data-centric reviewers punish leaderboard-only wins and
un-checkable discovery claims, and the whole evaluation must fit inside the 10-page
all-inclusive cap.
Define the mining task before the metric
- State the task operationally: inputs, outputs, and what a correct answer is. "Anomaly
detection" is a genre; "rank edges by anomalousness in a one-pass stream, evaluated against
injected ground truth" is a task.
- Fix the evaluation protocol — splits, negatives, thresholds, ranking cutoffs — before
running anything, and describe it precisely enough to reproduce inside the page cap.
The four evidence axes
| Axis |
Question it answers |
Typical evidence |
| Quality |
Is the mining result good on the task? |
Ranking/accuracy vs baselines with variance |
| Scale |
Does the scale claim hold? |
Latency/memory curves across data sizes |
| Mechanism |
Is the named mechanism the reason? |
Ablations toggling exactly that component |
| Validity |
Is the finding real, not an artifact? |
Controlled injections, known-truth checks |
A strong ICDM paper touches all four; missing "mechanism" or "validity" is the usual reason a
methodologically fine paper reads as thin.
Baselines and tuning symmetry
- Compare against current strong baselines, and tune them with the same budget you gave
your method; an under-tuned baseline is the fastest way to lose reviewer trust.
- Include the obvious simple baseline. If a cheap method nearly matches you, say so and argue
the regime where your mechanism pays off.
- Report every number with variance over seeded runs; a single-run table invites the "is this
noise?" review.
Ablations that isolate the mechanism
The mechanism-attached novelty of icdm-writing-style must be demonstrated, not asserted.
Mechanism: single-pass isolation sketch with m random partitions.
Ablation grid:
- remove the sketch, keep full storage -> isolates the streaming contribution
- vary m (partition count) -> maps the accuracy/memory knob
- swap the hash family -> tests sensitivity to the mechanism's core
- replace isolation score with density score -> isolates the isolation principle
Each row answers "was THIS the reason it worked?"
Test the scale claim, do not assert it
- If you claim scalability, plot behavior across at least an order of magnitude of data size,
and report the cost model (linear, sub-linear memory, amortized constant update).
- Separate wall-clock from asymptotic claims; hardware-dependent speedups need the hardware
stated and, ideally, an operation count that is not hardware-dependent.
Discovery validity: the ICDM instinct
- Where truth is unknown, build a setting where it is: injected anomalies, planted patterns,
synthetic graphs with known structure — so you can show the method recovers known signal.
- Guard against leakage: temporal tasks need time-respecting splits; graph tasks need to avoid
train/test edge overlap. State the guard explicitly.
- Do not overclaim when differences are within variance; an honest "matches at lower cost" is
stronger here than a fragile "outperforms."
Vignette: an ablation that saved the claim
A team reports strong stream-anomaly numbers but reviewers cannot tell whether the sketch or
the underlying isolation criterion did the work. Adding two ablation rows — full-storage
isolation (isolating the streaming contribution) and a density-score swap (isolating the
isolation principle) — showed the sketch preserved batch quality while the isolation principle
drove detection. The claim survived because the mechanism was shown, not stated, and both
rows fit in the appendix inside the 10-page cap.
Output format
[Task] <operational task definition>
[Axes covered] quality / scale / mechanism / validity - list gaps
[Baselines] strong + tuned symmetrically: yes / no
[Ablation] isolates the named mechanism: yes / no
[Validity] known-truth or leakage-guarded: yes / no
[Top evidence gap] <single most important missing experiment>
1---2name: icdm-experiments3description: Use when designing or auditing the empirical evaluation for an ICDM (IEEE International Conference on Data Mining) paper - mining-task definition, strong and fairly-tuned baselines, ablations that isolate the named mechanism, scalability curves that test scale claims, and discovery-validity checks that separate real findings from evaluation artifacts.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/icdm-experiments6---78# ICDM Experiments910Design the evaluation an ICDM reviewer will trust: a defined mining task, baselines tuned as11carefully as your method, ablations that isolate the mechanism, a measured scale story, and a12discovery-validity argument. ICDM's data-centric reviewers punish leaderboard-only wins and13un-checkable discovery claims, and the whole evaluation must fit inside the 10-page14all-inclusive cap.1516## Define the mining task before the metric1718- State the task operationally: inputs, outputs, and what a correct answer is. "Anomaly19 detection" is a genre; "rank edges by anomalousness in a one-pass stream, evaluated against20 injected ground truth" is a task.21- Fix the evaluation protocol — splits, negatives, thresholds, ranking cutoffs — before22 running anything, and describe it precisely enough to reproduce inside the page cap.2324## The four evidence axes2526| Axis | Question it answers | Typical evidence |27|---|---|---|28| Quality | Is the mining result good on the task? | Ranking/accuracy vs baselines with variance |29| Scale | Does the scale claim hold? | Latency/memory curves across data sizes |30| Mechanism | Is the *named mechanism* the reason? | Ablations toggling exactly that component |31| Validity | Is the finding real, not an artifact? | Controlled injections, known-truth checks |3233A strong ICDM paper touches all four; missing "mechanism" or "validity" is the usual reason a34methodologically fine paper reads as thin.3536## Baselines and tuning symmetry3738- Compare against current strong baselines, and tune them with the **same budget** you gave39 your method; an under-tuned baseline is the fastest way to lose reviewer trust.40- Include the obvious simple baseline. If a cheap method nearly matches you, say so and argue41 the regime where your mechanism pays off.42- Report every number with variance over seeded runs; a single-run table invites the "is this43 noise?" review.4445## Ablations that isolate the mechanism4647The mechanism-attached novelty of `icdm-writing-style` must be *demonstrated*, not asserted.4849```text50Mechanism: single-pass isolation sketch with m random partitions.51Ablation grid:52 - remove the sketch, keep full storage -> isolates the streaming contribution53 - vary m (partition count) -> maps the accuracy/memory knob54 - swap the hash family -> tests sensitivity to the mechanism's core55 - replace isolation score with density score -> isolates the isolation principle56Each row answers "was THIS the reason it worked?"57```5859## Test the scale claim, do not assert it6061- If you claim scalability, plot behavior across at least an order of magnitude of data size,62 and report the cost model (linear, sub-linear memory, amortized constant update).63- Separate wall-clock from asymptotic claims; hardware-dependent speedups need the hardware64 stated and, ideally, an operation count that is not hardware-dependent.6566## Discovery validity: the ICDM instinct6768- Where truth is unknown, build a setting where it is: injected anomalies, planted patterns,69 synthetic graphs with known structure — so you can show the method recovers *known* signal.70- Guard against leakage: temporal tasks need time-respecting splits; graph tasks need to avoid71 train/test edge overlap. State the guard explicitly.72- Do not overclaim when differences are within variance; an honest "matches at lower cost" is73 stronger here than a fragile "outperforms."7475## Vignette: an ablation that saved the claim7677A team reports strong stream-anomaly numbers but reviewers cannot tell whether the sketch or78the underlying isolation criterion did the work. Adding two ablation rows — full-storage79isolation (isolating the streaming contribution) and a density-score swap (isolating the80isolation principle) — showed the sketch preserved batch quality while the isolation principle81drove detection. The claim survived because the mechanism was *shown*, not stated, and both82rows fit in the appendix inside the 10-page cap.8384## Output format8586```text87[Task] <operational task definition>88[Axes covered] quality / scale / mechanism / validity - list gaps89[Baselines] strong + tuned symmetrically: yes / no90[Ablation] isolates the named mechanism: yes / no91[Validity] known-truth or leakage-guarded: yes / no92[Top evidence gap] <single most important missing experiment>93```