Data cleaning
Real data is dirty: missing, duplicated, mistyped, inconsistent. Cleaning
is not mechanical; every choice (drop this, impute that, cap the outlier)
is a decision that can preserve or destroy signal, and some cleaning steps
leak the future into the past. Clean deliberately, documenting why.
Method
- Understand before you clean. Do EDA first (see
exploratory-data-analysis) so you know what "dirty" means here and why:
cleaning blind (drop all rows with any null, cap everything at the 99th
percentile) destroys signal you did not know was there.
- Handle missing values by their meaning. Missing-at-random can be
imputed (median for skewed numerics, a constant plus a "was-missing"
indicator flag often beats fancy imputation); missing-with-meaning (no
subscription = null plan) is its own category, not a value to fill.
Dropping rows is fine when few and random, dangerous when missingness
correlates with the target (see the missingness discussion in
feature-engineering-tabular).
- Treat outliers as errors or extremes, deliberately. Confirmed data
errors (impossible values, typos) get fixed or removed; genuine extremes
get kept, capped, or transformed depending on the model's sensitivity
(tree models tolerate outliers; linear and distance-based do not).
Never silently delete extremes that are real data (see
exploratory-data-analysis step 3).
- Fix types and standardize. Parse dates as dates, numbers as numbers
(strip currency symbols, thousands separators), and standardize
categorical values (unify "USA"/"US"/"United States", trim whitespace,
normalize case). Inconsistent categories fragment signal across
duplicate levels.
- Deduplicate carefully. Find true duplicates (same entity recorded
twice) versus legitimate repeats (same value, different events). Dedupe
on the keys that define uniqueness, not blindly on all columns, and know
which record to keep (see idempotency thinking in data-pipeline-design).
- Fit cleaning on train only, apply to all. Imputation values, scaling
parameters, and category mappings are learned from the training data
and applied to validation/test, never fit on the full dataset. Fitting a
median-imputer on all data before splitting leaks test information into
training (see train-test-discipline, feature-engineering-tabular).
Boundaries
- Cleaning transforms data; log what you changed and why, because the
cleaning decisions are part of the analysis and affect the result. "We
dropped the outliers" is a choice a reviewer must be able to see and
question.
- Over-cleaning (aggressive outlier removal, heavy smoothing) can erase the
very signal or rare-but-important cases you care about (fraud, failures).
- Cleaning does not fix a biased or unrepresentative sample; a spotless
dataset drawn wrong is still wrong (see the sampling caution in
statistical-inference, correlation-causation).
1---2name: data-cleaning3description: Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.4---56# Data cleaning78Real data is dirty: missing, duplicated, mistyped, inconsistent. Cleaning9is not mechanical; every choice (drop this, impute that, cap the outlier)10is a decision that can preserve or destroy signal, and some cleaning steps11leak the future into the past. Clean deliberately, documenting why.1213## Method14151. **Understand before you clean.** Do EDA first (see16 exploratory-data-analysis) so you know what "dirty" means here and why:17 cleaning blind (drop all rows with any null, cap everything at the 99th18 percentile) destroys signal you did not know was there.192. **Handle missing values by their meaning.** Missing-at-random can be20 imputed (median for skewed numerics, a constant plus a "was-missing"21 indicator flag often beats fancy imputation); missing-with-meaning (no22 subscription = null plan) is its own category, not a value to fill.23 Dropping rows is fine when few and random, dangerous when missingness24 correlates with the target (see the missingness discussion in25 feature-engineering-tabular).263. **Treat outliers as errors or extremes, deliberately.** Confirmed data27 errors (impossible values, typos) get fixed or removed; genuine extremes28 get kept, capped, or transformed depending on the model's sensitivity29 (tree models tolerate outliers; linear and distance-based do not).30 Never silently delete extremes that are real data (see31 exploratory-data-analysis step 3).324. **Fix types and standardize.** Parse dates as dates, numbers as numbers33 (strip currency symbols, thousands separators), and standardize34 categorical values (unify "USA"/"US"/"United States", trim whitespace,35 normalize case). Inconsistent categories fragment signal across36 duplicate levels.375. **Deduplicate carefully.** Find true duplicates (same entity recorded38 twice) versus legitimate repeats (same value, different events). Dedupe39 on the keys that define uniqueness, not blindly on all columns, and know40 which record to keep (see idempotency thinking in data-pipeline-design).416. **Fit cleaning on train only, apply to all.** Imputation values, scaling42 parameters, and category mappings are learned from the training data43 and applied to validation/test, never fit on the full dataset. Fitting a44 median-imputer on all data before splitting leaks test information into45 training (see train-test-discipline, feature-engineering-tabular).4647## Boundaries4849- Cleaning transforms data; log what you changed and why, because the50 cleaning decisions are part of the analysis and affect the result. "We51 dropped the outliers" is a choice a reviewer must be able to see and52 question.53- Over-cleaning (aggressive outlier removal, heavy smoothing) can erase the54 very signal or rare-but-important cases you care about (fraud, failures).55- Cleaning does not fix a biased or unrepresentative sample; a spotless56 dataset drawn wrong is still wrong (see the sampling caution in57 statistical-inference, correlation-causation).
Amey-Thakur