Data Leakage Hunting

Data Leakage Hunting

Every other defect makes something fail. This one makes everything look better — which means the normal feedback loop of engineering runs in reverse. A leak is discovered by celebration, adopted into the pipeline, published in the README, and found six months later by production.

So the alarm has to be inverted, and it is the first rule of this skill:

An unexpectedly good result is a bug report until proven otherwise.

The invariant being violated is always the same one, and it is worth writing on the wall:

No information derived from the evaluation data may influence anything used to produce the prediction. Established at the split (T0), it must still hold at fit time, at feature-construction time, at tuning time, and at the moment the number goes in the report (Tn).

That is invariant-hunting's shape exactly: a property established in one place that has to survive a journey. The difference is that here the violation rewards you, so nobody goes looking.

Composes with the library:

• invariant-hunting — the T0→Tn frame; this is its ML instance
• model-evaluation-discipline — the number the leak corrupts, and the baselines that make a leak visible
• training-run-provenance — the split must be an artifact, not a seed
• discriminating-proof — the falsification experiments in Part 3
• honest-degradation — a leaked metric is the archetype of a plausible, confident, wrong answer
• daubert-defensible-writing — how to report that every prior number is void

Part 1 — Write the split contract before you model

Most leaks are decided before a single model is trained, by three questions nobody wrote down. Answer them in the repo, not in your head:

1. What is the unit of generalization? The thing the model must work on next, that it has never seen: a new user, a new patient, a new document, a new day, a new hospital. If the unit is a user and you split on rows, the evaluation is measuring memorization of users it already knows.
2. What is the time boundary? If the data has a time dimension and the model will be used to predict forward, then every evaluation row must be strictly after every training row. A random split on time-ordered data lets the model interpolate a future it will never have.
3. What defines a group? Same entity, same session, same source document, near-duplicate image, augmented copies of one original, retried request. All copies of one group belong on one side of the split.

Then store the split as data — the actual ids or indices, hashed and versioned — not as the seed that generated it. A seed is not a split: change a library version and the same seed produces a different partition, and every comparison you made becomes meaningless (training-run-provenance).

Part 2 — The eight families

Name the family before testing. Each has a distinct habitat and a distinct falsifier.

1. Preprocessing fitted before the split. A scaler, imputer, encoder, vocabulary, PCA, feature selector, or normalization statistic computed over the full dataset. The single most common leak in existence, and it is one line of code order. The fix is structural: the entire transformation chain lives inside a pipeline object that is fitted on the training fold only — and inside each fold in cross-validation, not once around it.
2. Temporal leakage. A random split on time-series data; a feature whose aggregation window extends past the prediction timestamp; a join that attaches a value known only later; a feature store queried "as of now" instead of "as of the event". Anything computed with groupby().mean() over the whole table is a candidate.
3. Group leakage. The same entity in both splits — patient, user, device, session, article, or several images from one photo series. The score measures recall of the entity, not generalization to a new one.
4. Target leakage in a feature. A column that is a consequence, proxy, or encoding of the label: account_closed_date for churn, n_treatments for diagnosis, a field only populated after the outcome, or an id that correlates with the class because of the order in which data was collected. The tell is a single feature with implausible importance.
5. Duplicates and near-duplicates across splits. Web-scraped corpora, augmented copies, retried records, boilerplate. Count them; do not assume.
6. Tuning on the test set. The slowest leak and the hardest to see: each experiment that peeks transfers a few bits of the test set into your choices. Threshold selection, early stopping, feature selection, architecture search, and "I tried it and it was worse" are all peeks. After a hundred experiments the test set is a training signal.
7. Label-process leakage. The annotator saw a model prediction; labels were derived from a source that is also a feature; two labellers' agreement was used to drop the hard examples. The test set is now easier than reality.
8. Pretraining contamination. The benchmark exists in the pretrained model's corpus. Standard for any public dataset and any foundation model — and it makes a public benchmark an upper bound, not a measurement.

Part 3 — Falsify, do not eyeball

Suspicion is cheap. These experiments are the discriminating kind — each one predicts a specific result, and the wrong result is the finding (discriminating-proof).

The label shuffle deserves its own line, because it is the negative control this whole field is missing: a model trained on randomized labels must not beat the baseline. If it does, stop everything — no result from that pipeline means anything until it is explained.

Run these before you report a good number, not after someone doubts it.

Part 4 — When you find one, the numbers are void

This is where the reporting discipline matters, because the temptation is to fix the leak, rerun, and quietly replace the number.

• Every metric produced by the leaking pipeline is void, not adjusted. They do not get corrected downward; they get withdrawn. Any decision made on them (this architecture beat that one, this feature helped) is unsupported and has to be re-run.
• Say it explicitly wherever the old number lives: README, dashboard, paper, ticket, slide. claim-provenance-discipline — a number that traveled keeps traveling until someone stops it by name.
• Fix it structurally. "We removed the ID column" is a point fix for one instance. The class fix is a pipeline that cannot express the leak: the transform chain fitted inside the fold, splits materialized as versioned artifacts, a feature store with point-in-time correctness, and a CI check that fails on cross-split id overlap.
• Add the falsifier as a permanent test. The label-shuffle control and the cross-split-overlap check belong in CI. They cost minutes and they are the only things that catch the next leak the day it is introduced.

Deliverable

## Leakage review — <dataset> / <pipeline> @ <version>
Unit of generalization: <entity>   Time boundary: <yes/no + cutoff>
Group key: <field>   Split stored as: <artifact hash, not a seed>

### Families checked
family | checked how | result | evidence

### Falsification runs
shuffled-labels: <score vs baseline>   10%-data: <score>
group-split: <delta>   time-split: <delta>   cross-split duplicates: <n>

### Findings
<id> — family — the leaking path — which reported numbers are now void

### Structural fixes + CI controls added
### Not checked

Anti-patterns

• Celebrating the jump instead of investigating it.
• Fitting the scaler, encoder, or vocabulary before the split — or once around a cross-validation loop instead of inside each fold.
• Storing the split seed rather than the split.
• Random splits on data with a time dimension or a repeating entity.
• Treating cross-validation as protection against leakage; it multiplies preprocessing leaks rather than preventing them.
• Reporting the improved number after fixing the leak, and leaving the old one in the README as if it had merely been superseded.
• Removing one leaking column and declaring the class closed.
• Never running the shuffled-label control, so the pipeline has never once been proven capable of producing a bad score.
• Reusing a public benchmark with a foundation model and reporting the result as a measurement rather than as an upper bound.